WaitFor doesn't seem to work
I use the self-hosted version of Firecrawl in a project where I scrape over 100 websites. I run into an issue with a specific website that uses a lot of Javascript. About half the time the markdown that Firecrawl outputs doesn't contain the information that I'm after (which is being loaded in by Javascript). The rest of the time it works fine and I get all the expected info in the markdown. I tried using a waitFor time of 10000 to force Firecrawl to wait for everything to load. However, I see that half the time it finishes after 6-8 seconds and the info I want is still not in the markdown.
Am I misinterpreting the waitFor parameter? Isn't it a minimum wait time?
Hope someone can explain this to me.. Thanks! 😅
2 Replies
As stated in the docs for the waitFor parameter: Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.
Can you give me the URL that causes you issues so that I can try to reproduce the issue and help it solve?
Yeah I interpreted that as: it will wait for the specified time before continuing, no matter what. But apparently it doesn't.
The url in question is https://vacatures.knmg.nl/ It's about the job listings on the right sight of the page. The rest of the main content is always present in the markdown, the stuff on the right only half of the time. The playground on the Firecrawl website consistently provides the right content, so I'm sure it must be possible to achieve the same result with the self-hosted version.