Performance: Sequential queue processing instead of parallel (horizontal scaling)
I recently converted a piece of logic into multiple workers, hoping to improve performance drastically by separating the logic in simple repeatable workers.
Im processing a list of data and passing this to the next worker using Queue [binding].
What i'm seeing as that the queue is being processed message per message instead of in parallel => concurrent => horizontal scaling.
I'm having a hard time figuring this out.
I've tried:
Im processing a list of data and passing this to the next worker using Queue [binding].
What i'm seeing as that the queue is being processed message per message instead of in parallel => concurrent => horizontal scaling.
I'm having a hard time figuring this out.
I've tried:
- seperate messages
- batches
- batch sizes
