Crawlee & Apify•2w ago

Hi,

Hi, I’m using APIFY to scrape reviews for a business that has around 500,000 reviews. I’m running the scraper daily, but I don’t want to re-fetch all 500K reviews every day. Most scrapers on APIFY don’t offer cursor, offset, or limit parameters. Is there any built-in mechanism in APIFY to handle this kind of incremental scraping? Or any recommended approach to avoid retrieving everything on each run? Thanks.

4 Replies

azzouzana•2w ago

How review available? (Each one has a separate URL)?

modern-teal•2w ago

Yes it's 1 separate URL usually.

MEE6•2w ago

@SUNSHINE just advanced to level 1! Thanks for your contributions! 🎉

azzouzana•2w ago

And you're developing the actor yourself or just using one ftom the store?

Gaming

Programming

Hi,

Did you find this page helpful?