Best way to process queue events in .NET without blocking or losing backpressure?
I’m working on a .NET microservice that consumes events from a queue (Kafka, Service Bus, etc.). The queue already guarantees ordering per key/partition, so ordering is not the problem. The issue is how to process messages concurrently without losing backpressure.
Problems I’m trying to solve:
If I await HandleAsync(...) directly in the consume loop → processing is slow, consumer builds lag.
If I Task.Run every message → unbounded concurrency, no backpressure, risk of overloading downstream services.
Bounded concurrency (SemaphoreSlim, fixed worker count, etc.) works but requires choosing a magic number (e.g., 5, 10, 20 workers) which doesn’t scale naturally.
What’s the recommended .NET pattern for scalable event consumption where:
Ordering per key/partition is preserved
Backpressure is maintained
No need for hardcoded internal concurrency limits
Scaling should ideally come from metrics like cpu load so its easier to setup via container apps (azure) or k8s
14 Replies
might i ask why are you worried this much about consuming resources? do you already know how many messages this will process? are these tasks in average intensive (so maybe you wouldn't have more than 1 task/core) or less so (so maybe 2+ tasks/core would be fine)?
Its important to me because when setting up multiple microservices that communicate via service bus or kafka or whatever async method i wanna make sure it scales seamlessly on azure; and a good battle tested pattern would be very helpful here
can you elaborate on what you mean by "maintaining backpressure"?
by "maintaining backpressure" do you just mean allowing the queue to fill up, if the upstream is generating events faster than the downstream can process them?
SignalR is what comes to my mind when it comes to asynchronous event processing. Not sure how it handles backpressure, though
either that or Channels
not sure if they fulfill the ordering requirement, though
they should fulfill the backpressure requirement
"scaling seamlesly" is a pretty big aim
what i meant by that is if i have it on some serverless deployment or k8s i dont want to worry about any magic numberes on a SemaphoreSlim u know...
i mean one of the simpler things is to just do Task.Run so consume it all like crazy.... and trigger scaling when cpu is at 60% lets say.... but that risks the idea if there is a huge surge of customers and if it doesnt have the time to rescale, could be an issue
backpressure: A mechanism that prevents producers from overwhelming consumers by slowing down or blocking the producer when the consumer can’t keep up.
this is already solved by kafka/service bus; here my focus is more on the consumer side as that seems more random to me
Why would you want backpressure? For you to keep back pressure but not build up more, you would need to auto scale your throughput according to the back pressure your seeing. You can do this to a degree, but I wouldnt not relie on such a system.
we can honestly drop the backpressure; given i will be using service bus in the cloud so we just produce how many messages we want
i think we have a xy-problem situation here, i dont really understand why you can "choose" how many messages you "want"
Unknown User•5d ago
Message Not Public
Sign In & Join Server To View
what if the load varies
this is purely a how to do it better question; trying to find out if there is a more efficient silver bullet way to implement consumers
so i can replicate over multiple microservices the same consumer logic
so i can replicate over multiple microservices the same consumer logic
Unknown User•5d ago
Message Not Public
Sign In & Join Server To View
yes but the idea here was that how to have a unified consumer setup in each microservice; because each microservice will consume for a service bus/queue
so i know if i use SemaphoreSlim, HandleAsync(...) OR Task.Run
or even better if there is some other set way of doing this
Task.Run would make no sense to me
but again i don't know if in the discussion come out what the tasks to run are
commands, events, queries consumed from a queue facilitated by azure service bus service; in a microservice architecture; meaning eahc microservice taht wants to react to events that are happening in the system would need to have this consumer funcitonality