Hey guys, Bruno here.
Hey guys, Bruno here.
I'm crawling a site of around 500 pages with scrape option on (asking for html and markdown) using your cloud platform.
For this specific site, seems that the crawl went too fast and made the site to go down. Most scraped HTML have as content "Error establishing a database connection".
Is there any way to configure the speed of the crawling to avoid this?
Thanks
1 Reply
and important not is that the "success" attribute that came with the scrape page is "true"