500 errors from Firecrawl Scrape API
I am trying to scrape anywhere between 2 and 25 webpages. I get errors every now and then like this:
[scrape_webpage] ERROR Response body: {"success":false,"code":"SCRAPE_SITE_ERROR","error":"Specified URL is failing to load in the browser. Error code: ERR_TUNNEL_CONNECTION_FAILED"}
Most pages on the same website go through, but rather consistently I get one with this error.
I can run the exact same URL and JSON schema through the Playground API and it works. I have tried using Stealth mode and sometimes this fixes it, but I am wondering if this is some form of bot protection by the website.
Any recommendations on how to prevent this? The 500 error does not show on the activity logs on the Firecrawl Admin so it almost feels like the issue is with the Firecrawl API itself.
[scrape_webpage] ERROR Response body: {"success":false,"code":"SCRAPE_SITE_ERROR","error":"Specified URL is failing to load in the browser. Error code: ERR_TUNNEL_CONNECTION_FAILED"}
Most pages on the same website go through, but rather consistently I get one with this error.
I can run the exact same URL and JSON schema through the Playground API and it works. I have tried using Stealth mode and sometimes this fixes it, but I am wondering if this is some form of bot protection by the website.
Any recommendations on how to prevent this? The 500 error does not show on the activity logs on the Firecrawl Admin so it almost feels like the issue is with the Firecrawl API itself.
