Channels usage for serial chunked async calls
I get a list of ids and I have one service I need to call in chunks in parallel and once a chunk is done start calling a second service with chunks in a different size also in parallel. I used channels but not sure if it is the correct approach or is there an easier way to do it (also not sure if write to channel needs to be async?)
12 Replies
Here is the code:
Unknown User•2d ago
Message Not Public
Sign In & Join Server To View
@TeBeCo So just let me make sure I get correctly:
1. Channels ARE the correct solution here?
2. After
GetService2Data
call instead of adding to the ConcurrentBag
send it to a new channel. The new channel will handle the "concurrency probelm" and I can just add to a list from the reader
3. If I should avoid IAsyncEnumerable
- what;s your alternative suggestion? Also why is it a bad idea? Mind explaining?
4. Should I use TryWrite
or WriteAsync
?
5. Currently the service 2 semaphore is used after adding to the list - maybe I should WaitAsync
inside the async enumreable? (unless of course I get rid of it)
Ah got it - you mean as there are multiple tasks writing to the results channel I need multiple producers in the results channel. Makes sense
So you mean to say (relating to my question number 2) that it is better the ConcurrentBag
in terms of performance?Unknown User•2d ago
Message Not Public
Sign In & Join Server To View
see
$channel
Unknown User•2d ago
Message Not Public
Sign In & Join Server To View
Opinion: first article looks better as an introduction than the devblog one, but but should still be read in that order
https://github.com/tkp1n/ndportmann.com/blob/master/posts/2019-01-03--system-threading-channels/index.md
https://web.archive.org/web/20221129052633/http://ndportmann.com/system-threading-channels/
https://devblogs.microsoft.com/dotnet/an-introduction-to-system-threading-channels
https://www.stevejgordon.co.uk/an-introduction-to-system-threading-channels
Video:
https://docs.microsoft.com/en-us/shows/on-net/working-with-channels-in-net
@TeBeCo I read the links - some of my questions still stand though
Also it seems IAsyncEnumerable is actually preferred in most cases if I understood correctly
@TeBeCo did some changes and will post again the code
But still waiting for some questions answered if you have an answer 🙂
Unknown User•22h ago
Message Not Public
Sign In & Join Server To View
@TeBeCo first of all sleeping is important and we can continue tomorrow 🙂
Concerning semaphore - I used it to limit the parallel calls. The thing is - I chunk the incoming items so I am not entirely sure if I need bounded channel or use semaphore to limit the reading amount or just read it all and let the semaphore and tasks list handle the limiting.
Anyway I guess the example I a bit complex. I will post an updates one and hopefully you will be more "sharp" tomorrow and could help.
Good night!
Btw that is the ReadAllAsync source (pretty similar):

Lastly here is an updated version - please let me know how does it seem tomorrow: