CA
Crawlee & Apify•3y ago
robust-apricot

Recover endless pagination items by clicking on showMore button

Hey guys ! Im trying to scrape all the products from a listing with an endless pagination. In order to load the other items, I've got to click on a showMore button. I looked it up on the docs and tried several syntaxes but I couldn't get it to work.. 
await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}
await page.$eval('main', (main) => {
    main.querySelector('.c-endless-paginator__button').click();
})
await page.waitForTimeout(5000);
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedUrls = [];
    $products.forEach(async ($product) => {
        await scrapedUrls.push({
            url: $product.querySelector('.c-product-list-item--grid__title').href
        });
    });
    return scrapedUrls;
});
for (let i = 0; i < products.length; i++) {
    await requestQueue.addRequest({
        url: products[i].url,
        userData: {
            depth: request.userData.depth + 1
        }
    })
}
I don't get why the products won't load even after the click and the waitForTimeout. Thank you for your help !
2 Replies
foreign-sapphire
foreign-sapphire•3y ago
This might help you, it can also click: https://crawlee.dev/api/playwright-crawler/namespace/playwrightUtils#infiniteScroll
playwrightUtils | API | Crawlee
A namespace that contains various utilities for Playwright - the headless Chrome Node API. Example usage: ```javascript import { launchPlaywright, playwrightUtils } from 'crawlee'; // Navigate to https://www.example.com in Playwright with a POST request const browser = await launchPlaywright(); c...
foreign-sapphire
foreign-sapphire•3y ago
+ do screenshots at the end of the scroll so you see what happens on the page

Did you find this page helpful?