Multiple queues
Is there a way to have a single crawler read from multiple queues?
2 Replies
dependent-tan•2y ago
Hi @kawaru, unfortunately, there is no direct solution for this.
Maybe you could try to create for each queue a separate basic crawler and only pass requests to the main crawler queue.
absent-sapphireOP•2y ago
Yeah, I couldn't find anything, either. So what I'm doing is rather than using enqueueLinks I'm passing the URLs out to another queue that the worker for that adds them to the default queue. Also having it removed the item from the queue since, from my tracing of the queue and other storage classes, items don't get removed unless they get purged.