Limit speed each request
Hi,
I dont know how the crawl works but it's possible to slow the crawl and dont get detected ?
I check the doc : limit is the total page crawl
poll_interval is the status of the crawl
We need to use proxies , PROXY_SERVER=
I just want to test one some sites but i dont want to get blocked with my ip
Thank
1 Reply
Hi @Jack74r, we have architected a lot of our systems to crawl and run scrapes as fast as possible, so we don't have a way to add deliberate delays, since this is not an issue we encounter on production.