CA
Crawlee & Apify•2y ago
vicious-gold

infiniteScroll and enqueueLinksByClickingElements

Hello Team, I'm trying to crawl a page that has lazy loaded images (on scroll) and an element on the first page that is a JS event 'button' that expands the pages of "posts" on the page. I'm trying to use the below code, however, it seems like the request never gets filled, the stats show 'requestsTotal:0'. 
async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}
async requestHandler({ request, page, enqueueLinks, enqueueLinksByClickingElements, infiniteScroll, log }) {

    // Extract links from the current page
    // and add them to the crawling queue.
    await enqueueLinks({ 
      ...<snip>
    });

    await enqueueLinksByClickingElements({
      page,
      selector: '.js-page-load',
      requestQueue: rQueue,
    });

    await infiniteScroll(page, { timeoutSecs: 3000, waitForSecs: 1000 });
    
}
I'm trying to target this: 
<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>
<button class="pagination-load js-page-load" data-href="/page/2/">Load More <span></span></button>
On the first page..so far it seems like this button is only on the first page. I'm also using preNavigationHooks to read the network requests and store image LINKS only. I don't know if this code should be in preNavigation hooks instead? Not sure. Thanks for your help as always.
1 Reply
vicious-gold
vicious-goldOP•2y ago
Ok well one problem is its seconds not milliseconds, so the queue not requesting having to wait, so I've updated that and running a test that is now in progress. also added method as playwrightUtils 
await playwrightUtils.enqueueLinksByClickingElements({
  page,
  selector: '.js-page-load',
  requestQueue: rQueue,
});
const InfiniteScrollOptions  = {
  timeoutSecs: 5,
  waitForSecs: 3,
};
await playwrightUtils.infiniteScroll(page, InfiniteScrollOptions );
await playwrightUtils.enqueueLinksByClickingElements({
  page,
  selector: '.js-page-load',
  requestQueue: rQueue,
});
const InfiniteScrollOptions  = {
  timeoutSecs: 5,
  waitForSecs: 3,
};
await playwrightUtils.infiniteScroll(page, InfiniteScrollOptions );

Did you find this page helpful?