Scraping not returning with FIRE-1
Hello,
I have a scraping task (all in python). My approach is to use the default scraper, review schema results, and if they aren't great, I try again using FIRE-1.
I'm using async in python to wrap a timeout around this, and I set a 120 second timeout, and the FIRE-1 results still are not returning. However, in the playground, I do see it work.
Any obvious things I should try before I debug further or provide more info / code-snippets, etc?
Thanks!
4 Replies
Hey! Which endpoint are you using and which version of the API?
hey @PBinPA
try these debugging steps:
1. Test FIRE-1 directly (skip the default scraper + schema check)
2. Increase timeout (ex, 300+ seconds)
@micah.stairs I'm not actually sure? I'm just creating a FirecrawlApp with my API key, and installed firecrawl via PIP. Package version is 4.3.6.
@harshithvh Strangely, it's all started working now. I'm not sure why. I did create a simple single test script that only did the search I wanted. That worked and took 53 seconds to execute (without using the default scraper first).
Then, I tried my more complex app, and it used the default scraper, recognized it didn't work, fell back to FIRE-1, and seems to have returned the correct data. So I don't know if this was a temporary issue or not.
Regardless, I'll increase default timeout to 300s.
I don't want to use FIRE-1 all the time, because the default scraper works just fine in 95% of the cases, and presumably is lighter weight, faster, cheaper than FIRE-1
hey @PBinPA
glad to hear that!
Just let me know if you have any further issues