INTERNAL SERVER ERROR
Hi team,
I have been getting this error more frequently when trying to scrape webpages using the /scrape endpoint.
Error information - Internal Server Error: Failed to scrape URL. All scraping methods failed for URL: https://www.atroposhealth.com/research-informatics/ - ['Request failed with status code 404', 'INTERNAL SERVER ERROR']
Some other error reasons observed: [WebSocket is not open: readyState 3 (CLOSED), Timed out]
Some other URLs which are throwing a similar error during scraping; for your reference:
1. https://sapienbio.co.in/assessment-of-dll3-and-cd3-expression-in-pediatric-tumor-ffpe-samples-for-potential-dll3-targeted-immunotherapy/
2. https://www.atroposhealth.com/research-informatics/
3. https://www.novadiscovery.com/glossary/in-silico-clinical-trial/
This is something which is not critical at the moment, but sharing here so that the team can further explore the root cause for this.
Or if you guys are already aware of this, please let me know.
TIA.
5 Replies
@Adobe.Flash @mogery for visibility.
Hi there, fixed the 404 one showing up as an error. The WebSocket one is tougher and generally indicates a fatally failed scrape, so that's a proper error you're seeing there. Thanks for reporting and including the URLs, we'll take a look! 🔥
also pinging @thomas on this one
Thanks @mogery for the quick response.
@Sachin I also applied further fixes to revert the crawl behaviour from last week. This was an issue in some code I added to try and improve error handling so we can catch faulty scrapes and fix them -- but it was a bit too aggressive.
@mogery glad to hear that. Thanks for the update.