Help with scraping a paginated website using Firecrawl Actions
Hey all! I'm new to Firecrawl. I have a website I'm looking to scrape that is paginated (think several "next" and "previous" buttons that javascript render different content without changing the URL).
How would I do it with actions? I'm familiar with the "wait" and "click", but how would I actually aggregate the scraped content for each paginated page? the "scrape" isn't working, but I'm probably using it wrong:
"actions": [
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='2']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='3']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='4']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
}
]
},
"scrape_params": {
"formats": ["markdown"],
"onlyMainContent": True,
"timeout": 30000
}
Any help would be appreciated!! Thanks
How would I do it with actions? I'm familiar with the "wait" and "click", but how would I actually aggregate the scraped content for each paginated page? the "scrape" isn't working, but I'm probably using it wrong:
"actions": [
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='2']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='3']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
},
{
"type": "click",
"selector": "a.pgn.page-numbers[data-page='4']"
},
{
"type": "wait",
"milliseconds": 3000
},
{
"type": "scrape"
}
]
},
"scrape_params": {
"formats": ["markdown"],
"onlyMainContent": True,
"timeout": 30000
}
Any help would be appreciated!! Thanks
