External Queue Provider
Hello, I started using Crawlee and am quite impressed with the speed and
anti-bot protection. However I do want to scale the system horizontally between different
machines (nodes) in cluster and therefore I need to have shared queue broker (like Redis or RabbitMQ).
Is it possible to configure Crawlee to use my own queue instead of local in-process one? Or There is no out-of-the-box
way and I should implement this myself listening to my Redis queue and adding to in-process queue myself?
3 Replies
other-emerald•3y ago
Well, I found a few attempts to implement crawlee with external message queue, see
https://discord.com/channels/801163717915574323/1056348705407651941
wise-whiteOP•3y ago
thanks, just did this with Redis, seem to be working
other-emerald•3y ago
any unexpected problems? error handling?
(I am fighting with error handling at the moment...)