408 / 502 ERRORs when scraping particular URLs via code/manually
Hello guys, I have developed a simple script that iterates through sitemap and scrapes each URL individually via the scrape endpoint.
However, some particular URLs just return 408 / 502 errors, when scrape endpoint is called, dunno why - but its not the issue in the code it seems, because even when I try to scrape them manually from the Firecrawl dashboard - nothing happens.
These URLs are in case of this particular client:
"https://icuk.cz/pro-region/projekt-storemore",
"https://icuk.cz/pro-region/rur/",
"https://icuk.cz/pro-region/projekt-integrage/",
"https://icuk.cz/pro-region/projekt-coeus/"
5 Replies
Here are some examples of the errors from error log:
I would be very greatful for any help...thank you
I'm also suddenly receiving 408 errors from Firecrawl
Hi @Pavel Čermák 🦾 @noli We had some issues and we've adjusted our machines. We're still monitoring everything but this should be resolved now.
Also @Pavel Čermák 🦾 Our
/crawl endpoint automatically reads sitemap data, so you could just use that instead of spawning scrapes manually