WaitFor doesn't seem to work
I use the self-hosted version of Firecrawl in a project where I scrape over 100 websites. I run into an issue with a specific website that uses a lot of Javascript. About half the time the markdown that Firecrawl outputs doesn't contain the information that I'm after (which is being loaded in by Javascript). The rest of the time it works fine and I get all the expected info in the markdown. I tried using a waitFor time of 10000 to force Firecrawl to wait for everything to load. However, I see that half the time it finishes after 6-8 seconds and the info I want is still not in the markdown.
Am I misinterpreting the waitFor parameter? Isn't it a minimum wait time?
Hope someone can explain this to me.. Thanks!
Am I misinterpreting the waitFor parameter? Isn't it a minimum wait time?
Hope someone can explain this to me.. Thanks!
