Crawlee & Apify•3y ago

Load more button with Puppeteer

I integrated following logic to click on the show-more button when there is one. But unfortunately It could not find the button... Also with page. waitforSelector(buttonSelector) it still does not work. Why nothing happens while clicking the button? An example for a product page you can find here (Click on SKUs): https://buildings.honeywell.com//us/en/products/by-category/access-control/control-panels/pro4200-controllers

export const handleProduct = async (page,url, crawler, dataset) => {
    page.waitForTimeout(4000);
    
    const buttonSelector = 'button#load_more_sku.secondary-cta.d-inline-block';
        while (true) {
            log.info('Waiting for the "Show more" button.');
            try {
                log.info('Clicking the "Show more" button.');
                await page.click(buttonSelector);
            } catch (err) {
                // Ignore the timeout error.
                log.info('Could not find the "Show more button", '
                    + 'we\'ve reached the end.');
                break;
            }
            
        }

export const handleProduct = async (page,url, crawler, dataset) => {
    page.waitForTimeout(4000);
    
    const buttonSelector = 'button#load_more_sku.secondary-cta.d-inline-block';
        while (true) {
            log.info('Waiting for the "Show more" button.');
            try {
                log.info('Clicking the "Show more" button.');
                await page.click(buttonSelector);
            } catch (err) {
                // Ignore the timeout error.
                log.info('Could not find the "Show more button", '
                    + 'we\'ve reached the end.');
                break;
            }
            
        }

PRO4200 Controllers | Security Control Panels | Security Controls |...

Learn all about the Honeywell Security PRO4200 Controllers. Click to find product details, documentation, ordering info and more.

5 Replies

MEE6•3y ago

@Deleted User just advanced to level 1! Thanks for your contributions! 🎉

afraid-scarlet•3y ago

I recommend you starting with extensive screenshoting to find the root cause. More here: https://developers.apify.com/academy/node-js/analyzing-pages-and-fixing-errors It might be blocking or wrong page or the button doesn't work. Sometimes only javascript click inside page.evaluate works

Apify

How to analyze and fix errors when scraping a website · Apify Devel...

Learn how to deal with random crashes in your web-scraping and automation jobs. Find out the essentials of debugging and fixing problems in your crawlers.

constant-blueOP•3y ago

Seems like a cookie is blocking the crawling process. How can I pass this or accept the policies of the website? This is the product site which I am scraping in the example: https://buildings.honeywell.com//us/en/products/by-category/video-systems/software/maxpro-nvr-software

MAXPRO® NVR Software | System Agreements & Upgrades | Security Cont...

Learn all about the Honeywell Security MAXPRO® NVR Software. Click to find product details, documentation, ordering info and more.

afraid-scarlet•3y ago

You can find the selector for the X close button and click on that after the page loads and you wait a bit. The cookie approval usually only appears with EU proxies

constant-blueOP•3y ago

I think found the button for the x close button but for some reason my logic still doesn't work and it still gets get's stuck at the point when the button appears.

export const handleProduct = async (page,url, crawler, dataset) => {
    page.waitForTimeout(4000);
    const reportingDataset = await Actor.openDataset('REPORTING');
    const storeId = Actor.getEnv().defaultKeyValueStoreId;
    const { actorId, actorRunId } = Actor.getEnv();
    const linkToRun = `https://console.apify.com/actors/actorId#/runs/actorRunId`;

    const buttonSelector = 'button#load_more_sku.secondary-cta.d-inline-block';
    const privacybutton ='button.close-pc-btn-handler'
        while (true) {
            log.info('Waiting for the "Show more" button.');
            page.waitForSelector(buttonSelector);
            try {
                    log.info('Clicking the "Show more" button.');
                    await page.click(buttonSelector);
                    page.waitForSelector(privacybutton);
                        if(privacybutton){
                            page.click(privacybutton);
                            log.info('Clicking the "Show more" button.');
                            await page.click(buttonSelector);
                        }

            } catch (error) { ...

export const handleProduct = async (page,url, crawler, dataset) => {
    page.waitForTimeout(4000);
    const reportingDataset = await Actor.openDataset('REPORTING');
    const storeId = Actor.getEnv().defaultKeyValueStoreId;
    const { actorId, actorRunId } = Actor.getEnv();
    const linkToRun = `https://console.apify.com/actors/actorId#/runs/actorRunId`;

    const buttonSelector = 'button#load_more_sku.secondary-cta.d-inline-block';
    const privacybutton ='button.close-pc-btn-handler'
        while (true) {
            log.info('Waiting for the "Show more" button.');
            page.waitForSelector(buttonSelector);
            try {
                    log.info('Clicking the "Show more" button.');
                    await page.click(buttonSelector);
                    page.waitForSelector(privacybutton);
                        if(privacybutton){
                            page.click(privacybutton);
                            log.info('Clicking the "Show more" button.');
                            await page.click(buttonSelector);
                        }

            } catch (error) { ...

Solution was that I had to firstable click on the SKU tab with await page.click(button), before clicking on the load more button.

Gaming

Programming

Load more button with Puppeteer

Did you find this page helpful?