Getting intermittent error page from sites ONLY when using Firecrawl
hello there!
i've successfully loaded the following site through other scrapers, including Playwright, which I know Firecrawl runs under the hood.
https://www.capptegy.com/events
I'm not sure if there's additional logging on the Firecrawl side of things, but I'm not seeing anything on the site logs and I've confirmed that they're not blocking any crawlers associated with Firecrawl IPs.
To further add some complexity to this, this site does not always return an error - it's intermittent.
Talking to their support team, I can confirm that this started happening on Tuesday of this week around 11:18pm Pacific time - I know there was a release from Firecrawl's side last Thursday, so I wonder if that's related.
I've also confirmed this is happening Client Side, so I'm not sure it's something on their end, but I'm looking for any help!
Here's a Crawl ID of my attempt to do this along with an image showing the error:
https://www.firecrawl.dev/playground/WcXQsklISZ73IrVb_dwO2
Please let me know if there are any questions i can help answer, but I'm hoping to get this resolved quickly!
i've successfully loaded the following site through other scrapers, including Playwright, which I know Firecrawl runs under the hood.
https://www.capptegy.com/events
I'm not sure if there's additional logging on the Firecrawl side of things, but I'm not seeing anything on the site logs and I've confirmed that they're not blocking any crawlers associated with Firecrawl IPs.
To further add some complexity to this, this site does not always return an error - it's intermittent.
Talking to their support team, I can confirm that this started happening on Tuesday of this week around 11:18pm Pacific time - I know there was a release from Firecrawl's side last Thursday, so I wonder if that's related.
I've also confirmed this is happening Client Side, so I'm not sure it's something on their end, but I'm looking for any help!
Here's a Crawl ID of my attempt to do this along with an image showing the error:
https://www.firecrawl.dev/playground/WcXQsklISZ73IrVb_dwO2
Please let me know if there are any questions i can help answer, but I'm hoping to get this resolved quickly!
