Search
Star
Feedback
Setup for Free
© 2026 Hedgehog Software, LLC
Twitter
GitHub
Discord
System
Light
Dark
More
Communities
Docs
About
Terms
Privacy
Firecrawl batch cannot crawl some urls - Firecrawl
F
Firecrawl
•
7mo ago
•
5 replies
Dylan Le
Firecrawl batch cannot crawl some urls
Hello everyone I am using firecrawl python sdk when I use batch
_scrape
_urls it can
't crawl
. Just 1 url is not successfully crawled it will logs error
. I wish it will ignore the error url and execute the remaining urls
. I have looked for the
"ignoreInvalidURLs
" attribute but can
't find it in python sdk
. here is my list of urls
:
[
'
https://www.britannica.com/biography/Stephen-Colbert
'
,
'
https://www.biography.com/movies-tv/stephen-colbert
'
,
'
https://www.cbs.com/shows/the-late-show-with-stephen-colbert/
'
,
'
https://www.imdb.com/name/nm0170306/bio/
'
,
'
https://en.wikipedia.org/wiki/Stephen_Colbert
'
,
'
https://en.wikipedia.org/wiki/The_Late_Show_with_Stephen_Colbert
'
,
'
https://www.televisionacademy.com/bios/stephen-colbert
'
,
'
https://en.wikipedia.org/wiki/List_of_awards_and_nominations_received_by_Stephen_Colbert
'
,
'
https://www.imdb.com/name/nm0170306/awards/
'
,
'
https://www.youtube.com/channel/UCMtFAi84ehTSYSE9XoHefig
'
]
Firecrawl
Join
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.
15,303
Members
View on Discord
Resources
ModelContextProtocol
ModelContextProtocol
MCP Server
Similar Threads
Was this page helpful?
Yes
No
Recent Announcements
Similar Threads
Firecrawl Crawl Issue - n8n Integration
F
Firecrawl / ❓┃community-help
4mo ago
Using firecrawl crawl inside Make.com
F
Firecrawl / ❓┃community-help
9mo ago
Firecrawl doesn't seem to crawl everything
F
Firecrawl / ❓┃community-help
2y ago
Firecrawl api not giving output for crawl jobs
F
Firecrawl / ❓┃community-help
2y ago