Issues and Inconsistencies During FireCrawl Testing
Hey everyone!
We’ve been testing FireCrawl for a bit at our company, and we’ve come across some issues that I wanted to get your thoughts on:
1. Crashes when viewing logs: We keep seeing crashes when trying to view crawl results on the logs page, with the error: "Application error: a client-side exception has occurred (see the browser console for more info)." It looks like a 504 Gateway Timeout. Is this something you guys are aware of? Any idea when it might be fixed?
2. waitFor impacting performance: On some websites, using waitFor seems to be the only way we can get valid results, but it really increases the crawl time. Has anyone else experienced this? Any tips for speeding things up? We’re seeing a lot of 200 status codes but with empty content.
3. Lack of detailed error info: When scraping fails, we only get generic errors like "500 INTERNAL ERROR" or "Request timed out". Is there any way to get more specific error details to help debug?
4. Inconsistent crawl results: We’ve noticed some inconsistencies when crawling the same site multiple times. Here are a couple of examples:
bootbarn.com:
id: b0d4744e-3cf9-4e30-bf6d-576530d8fc87: Had 60 errors (pages returning 500 INTERNAL ERROR), but the same pages scraped fine on a different run (e.g., https://www.bootbarn.com/sale/promotions/denim-deals/).
id: d8bd1cee-d469-4d2f-bf6f-c1f9a6fad863: No errors at all.
delaval.com:
id: 975ba184-8d29-473e-adfe-73174ebe0a57 and id: bae8c29a-bd92-47c9-925e-894529ced16c: We got a ton of timeouts (496 in one run, 510 in the next), but when we scrape these pages one by one, they work fine (e.g., https://www.delaval.com/en-gb/animal-welfare/cow-comfort-stalls/light-installation/).
Also noticed inconsistent page counts for the same site:
Wikipedia (Bicycle page):
id: d34788dd-d2d5-4252-8ba9-76524ff4a32c: 286 pages scraped.
id: 47a9224b-e75b-4c57-ad3e-e2cb7a33d1e6: 318 pages scraped.
Would love to hear any advice or thoughts on these issues!
Thanks in advance!
We’ve been testing FireCrawl for a bit at our company, and we’ve come across some issues that I wanted to get your thoughts on:
1. Crashes when viewing logs: We keep seeing crashes when trying to view crawl results on the logs page, with the error: "Application error: a client-side exception has occurred (see the browser console for more info)." It looks like a 504 Gateway Timeout. Is this something you guys are aware of? Any idea when it might be fixed?
2. waitFor impacting performance: On some websites, using waitFor seems to be the only way we can get valid results, but it really increases the crawl time. Has anyone else experienced this? Any tips for speeding things up? We’re seeing a lot of 200 status codes but with empty content.
3. Lack of detailed error info: When scraping fails, we only get generic errors like "500 INTERNAL ERROR" or "Request timed out". Is there any way to get more specific error details to help debug?
4. Inconsistent crawl results: We’ve noticed some inconsistencies when crawling the same site multiple times. Here are a couple of examples:
bootbarn.com:
id: b0d4744e-3cf9-4e30-bf6d-576530d8fc87: Had 60 errors (pages returning 500 INTERNAL ERROR), but the same pages scraped fine on a different run (e.g., https://www.bootbarn.com/sale/promotions/denim-deals/).
id: d8bd1cee-d469-4d2f-bf6f-c1f9a6fad863: No errors at all.
delaval.com:
id: 975ba184-8d29-473e-adfe-73174ebe0a57 and id: bae8c29a-bd92-47c9-925e-894529ced16c: We got a ton of timeouts (496 in one run, 510 in the next), but when we scrape these pages one by one, they work fine (e.g., https://www.delaval.com/en-gb/animal-welfare/cow-comfort-stalls/light-installation/).
Also noticed inconsistent page counts for the same site:
Wikipedia (Bicycle page):
id: d34788dd-d2d5-4252-8ba9-76524ff4a32c: 286 pages scraped.
id: 47a9224b-e75b-4c57-ad3e-e2cb7a33d1e6: 318 pages scraped.
Would love to hear any advice or thoughts on these issues!
Thanks in advance!
