Handling pagination within Apify task?
I've created a Playwright scraper task in the Apify web app.
I'm trying to sort out how to handle pagination that -
* uses a link bar bar (instead of lazy loading items)
* doesn't show the last page number until you've navigated within a few pages of end
* uses <button> elements for page links (instead of anchor tags with hrefs)
I want to navigate all of these pages to collect URLs that will be queued for scraping.
The docs mention there's a way to collect paginated URLs using the
Pseudo URLs (https://docs.apify.com/tutorials/apify-scrapers/web-scraper#pagination) but don't go into any detail on how to do this. I'm also guessing this approach won't work without an a[href] element.
What would be best way to handle this kind of pagination?
Thanks for any help.
---Apify
Web Scraper · Apify Documentation
Learn how to scrape a website using Apify's Web Scraper. Build an actor's page function, extract information from a web page and download your data.
2 Replies
rival-black•3y ago
You can try to catch request that is browser sending after clicking on the next button and then enqueue that request.
fair-roseOP•3y ago
Thanks @HonzaS I'll dig into that to see what I can sort out