Concurrent crawlers or maxRequests per Queue?
I'm crawling many websites everyday.
The ideal approach for me would be to have a maxRequestsPerMinute dependent on the website.
That way I'd have the crawler going at full speed, but crawling different pages from different websites in order to not surpass the websites request limit.
I don't think that's possible though. So how could I achieve this?
Should I run many crawlers at the same time? The problem with that is that it will require much more memory capacity, right?
1 Reply
optimistic-gold•3y ago
Well - actually, if you really need to limit the requests per minute, running several crawlers while allocating less memory for them will naturally slow it done + you could limit number of requests per minute. But yes - that's probably the way 👍