F
Firecrawl•7d ago
danny

Getting intermittent error page from sites ONLY when using Firecrawl

hello there! i've successfully loaded the following site through other scrapers, including Playwright, which I know Firecrawl runs under the hood. https://www.capptegy.com/events I'm not sure if there's additional logging on the Firecrawl side of things, but I'm not seeing anything on the site logs and I've confirmed that they're not blocking any crawlers associated with Firecrawl IPs. To further add some complexity to this, this site does not always return an error - it's intermittent. Talking to their support team, I can confirm that this started happening on Tuesday of this week around 11:18pm Pacific time - I know there was a release from Firecrawl's side last Thursday, so I wonder if that's related. I've also confirmed this is happening Client Side, so I'm not sure it's something on their end, but I'm looking for any help! Here's a Crawl ID of my attempt to do this along with an image showing the error: https://www.firecrawl.dev/playground/WcXQsklISZ73IrVb_dwO2 Please let me know if there are any questions i can help answer, but I'm hoping to get this resolved quickly!
No description
1 Reply
Gaurav Chadha
Gaurav Chadha•7d ago
@danny use the Stealth mode and disable parse pdf, it will work then. if you're using sdk or curl set this payload 
{
    "url": "https://www.capptegy.com/events",
    "sitemap": "include",
    "crawlEntireDomain": true,
    "limit": 10,
    "scrapeOptions": {
        "onlyMainContent": false,
        "maxAge": 172800000,
        "proxy": "stealth",
        "parsers": [],
        "formats": [
            "markdown"
        ]
      },
    "origin": "website"
}
{
    "url": "https://www.capptegy.com/events",
    "sitemap": "include",
    "crawlEntireDomain": true,
    "limit": 10,
    "scrapeOptions": {
        "onlyMainContent": false,
        "maxAge": 172800000,
        "proxy": "stealth",
        "parsers": [],
        "formats": [
            "markdown"
        ]
      },
    "origin": "website"
}
also, for the detail of the issue: The website https://www.capptegy.com/events. is using intelligent bot detection specially on the /events path which is blocking and throwing out error with captcha and if the error happens intermittently, also add delays. Hope this clears and resolves the issue :)
No description
No description

Did you find this page helpful?