Help with scraping a paginated website using Firecrawl Actions
Hey all! I'm new to Firecrawl. I have a website I'm looking to scrape that is paginated (think several "next" and "previous" buttons that javascript render different content without changing the URL).
How would I do it with actions? I'm familiar with the "wait" and "click", but how would I actually aggregate the scraped content for each paginated page? the "scrape" isn't working, but I'm probably using it wrong:
"actions": [
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='2']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='3']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='4']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
}
]
},
"scrape_params": {
"formats": ["markdown"],
"onlyMainContent": True,
"timeout": 30000
}
Any help would be appreciated!! Thanks
1 Reply
You can probably just use /crawl instead of /scrape. But if this site needs /scrape for whatever reason, then this looks fine. The data should be part of the response (but not in the normal spot since we need to return an array instead of a single result).