C#C
C#•13mo ago
Cryy

Asynchronous batch processing (queue) in C#

Hey guys, I have some batches to process and to do so I have to send each of my batches to external APi to process. I have no experience with async programming so I simplified it a bit by using Parallel.Foreach that create thread/task for each API calling and waiting for response. The reason I dont want to call API synchronously Is that processing of one batch can take a one sec or 30min (max). With the Parallel.Foreach solution it looks it works fine but form what I read on internet it's not a good approach so I want to implement kind of async queue that will take a few items from my collection of batches starts processing and after some of them is finished replace it with another etc. I tried to came up with something using ChatGPT but it only process specified number of batches (3 in my case) and than it does nothing but even if the code is short there are lot's of new concepts for me so I have no idea how to fixed this. 
public async Task ProcessBatchQueueAsync(IEnumerable<int> batchIds)
{
    var semaphore = new SemaphoreSlim(3); 
    var tasks = new List<Task>();

    foreach (var batchId in batchIds)
    {
        await semaphore.WaitAsync();

        var task = Task.Run(async () =>
        {
            try
            {
                await ProcessSingleBatchAsync(batchId);
            }
            finally
            {
                semaphore.Release();
            }
        });

        tasks.Add(task);
    }

    await Task.WhenAll(tasks);
}

private async Task ProcessSingleBatchAsync(int batchId)
{
    _loggerController.Log(LogLevel.Info, $"Processing batch {batchId}...");
    await Task.Delay(2000);
    _loggerController.Log(LogLevel.Info, $"Finished processing batch {batchId}...");
}
Was this page helpful?