No failure reason for crawl job
I am testing out firecrawl for one of our projects but I am finding that the crawl is repeatedly failing but no reason is given for why it failed. I only see this :
4 Replies
WARNING:root:Exception Crawl job failed or was stopped. Status: failed
Here is my code:
crawl_url = 'https://<url goes here in my actual code'
params = {
'crawlerOptions': {
'limit': 300,
'maxDepth': 300,
},
'pageOptions': {
'onlyMainContent': True,
'parsePDF': False
}
}
try:
crawl_result = app.crawl_url(crawl_url, params)
except Exception as e:
logging.warning(f"Exception {e}")
Hey @Janice , could you share the url you were trying so we can investigate? Feel free to dm me if you dont want to share publically
Thank you for replying Nick! I'm not sure what happened but things are working this morning. Actually things just stopped working again. :-(. My URL is: https://lsu.edu/urec
Other websites and urls within this website seem to be working
Hey, this was probably caused by the downtime we had earlier. I just tried this and it worked. sorry for the trouble!