Website with dynamically generated content using Javascript
The website is using a table that is generated by Javascript and when I scrape the website, it only grabs the default set of rows that is generated. When I update the table, the URL doesn't change so the scrapeURL function only picks up the first subset.
12 Replies
Can anyone help here?
Hey @Professor Meta, sorry for the late response
We cant access dynamic content that requires particular actions (like scroll and clicking). However, we can
access content that will appear on the page after some time with the waitFor parameter
Ah okay, yeah it requires a click to get to the next set of rows
This is something we may support soon. Keep an eye on our releases for more
Ok will do, otherwise great product!!
Any updates on this?
Here is an example
hey @Professor Meta! You can use our new Actions API for this. https://www.firecrawl.dev/blog/scrape-job-boards-firecrawl-openai
Firecrawl
Scraping Job Boards Using Firecrawl Actions and OpenAI
A step-by-step guide to scraping job boards and extracting structured data using Firecrawl and OpenAI.
oh amazing, I'll check it out
TY for the follow up
Hmmm, not seeing how actions is being used in the example above... the link to Actions also just points to the homepage, but I found this (https://docs.firecrawl.dev/introduction#interacting-with-the-page-with-actions)..
The page I'm dealing with is: https://onepiecetopdecks.com/deck-list/japan-op-09-the-new-emperor-decks/
And I need the Actions API to click and scroll through the rows of the table. Where can I find more details about the Actions API to potentially do this?
Firecrawl Docs
Quickstart | Firecrawl
Firecrawl allows you to turn entire websites into LLM-ready markdown
I have a similar issue. Did you find any solution @Professor Meta ?
Not yet, haven't had a chance to try out the Actions API, have you?