Crawlee & Apify•3y ago

Ability to change scraping speed and concurrency while the script is running

It would be nice to be able to change the scraping speed while running the crawler if you determine it is running too fast for the site to keep up, without having to stop and start the crawler again. It could be done from the crawlee CLI for example.

2 Replies

crude-lavender•3y ago

It is possible to modify the properties on AutoscaledPool within a route.

router.addHandler('some-label', async ({ crawler }) => {
    crawler.autoscaledPool.maxConcurrency = 10;
    crawler.autoscaledPool.desiredConcurrency = 5;
// ...

router.addHandler('some-label', async ({ crawler }) => {
    crawler.autoscaledPool.maxConcurrency = 10;
    crawler.autoscaledPool.desiredConcurrency = 5;
// ...

fascinating-indigoOP•3y ago

Thanks

Gaming

Programming

Ability to change scraping speed and concurrency while the script is running

Did you find this page helpful?