How can I get more data when the site is only providing 50 items per page then 40 pages per seller?
Good day guys
How can I get more data when the site is only providing 50 items per page then 40 pages per seller? So the scrapper is only getting 2k items in total when the seller has 7554 items in total. TIA
5 Replies
unwilling-turquoise•3y ago
You need to iterate through the search filters, see https://developers.apify.com/academy/advanced-web-scraping/scraping-paginated-sites
Apify
Scraping paginated sites · Apify Developers
Learn how to extract all of a website's listings even if they limit the number of results pages. See code examples for setting up your scraper.
rival-blackOP•3y ago
@Lukas Krivka hey thank you!!! Life saver!!!!
@Lukas Krivka I got the code working. Great!!! Thank you
exotic-emerald•3y ago
Twiitter is not paginated-- user scroll refresh. The Twitter Url Task only seems to return a maximum of 211 tweets per conversation even if the tweet has many more replies (all accessible from a single URL.). How to get more than 211 in this instance? Thnx
@BigChase just advanced to level 1! Thanks for your contributions! 🎉
unwilling-turquoise•3y ago
Scrolling based flows are much harder. Generally, they require login to go further