Trouble extracting URLs from a JavaScript-based paginated search
Hi everyone,
I'm struggling with scraping a list of URLs from a search results page. The site uses JavaScript to load 20 results at a time, and to get the next 20, I have to click a "Next" button. There’s no change in the URL or any query parameters that I can use to trigger pagination directly.
I was thinking about using actions in Firecrawl to simulate clicking through the pages and extracting the links, but I’m not sure if that’s the right approach or the most efficient one.
Has anyone dealt with something similar?
Should I try automating this with Firecrawl actions?
Or is there a better way to handle JavaScript-driven pagination when scraping?
Any tips or best practices would be greatly appreciated!
Thanks in advance 🙌
1 Reply