How to make queues pick up messages
How to make queues pick up messages faster?
Currently I see a few seconds lag from message being inserted in queue, vs being picked up.
This is critical for UX. I want to reduce this time. Will it be better if I shift to a workflow ?
5 Replies
I've noticed this as well - I've set
max_batch_size to 1 and max_batch_timeout to 0 in my consumer Worker and I'm still seeing 10-15 seconds of delay between writing and consuming it.I see <= 6s using http pull
Oct 1 21:58:26 -> 17:58:32
yeah next one was 56->59. basically instant (clock drift, polling, etc.) probably closer to 1s realtime
that's e2e too: so going from a remote server halfway across the world -> worker -> queue -> my server.
It has been discussed earlier in the channel, and I think the conclusion was that currently queues are not well suited for low latency use cases. I think there might be some improvements coming in this area, but no clue when that might happen.
Did anyone here try workflows instead ?
I can have a one step workflow as I need to maintain object state (cannot be durable).
The 4 second pause is killing my UX.
Max batch timeout doesn't make sense if max_batch_size is 1.
As soon as there is 1 item in the queue, the consumer will be triggered.
Correct. I started with just
max_batch_size, it didn't really work, and I added timeout just in case.