C#•3y ago

ASP NET Core 6 Launch without Debugging doesn't block Background Service but Debug Does

Hi, I'm trying to have a ASP.NET Core BackgroundService run really in background, i mean really don't interrupt with request/response part and I've been trying many stuff

    private Thread? _workerThread;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        Logger.LogInformation("Monitoring Event Triggering Start");
        _workerThread = new Thread(() =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                if (EventQueue.Count == 0) continue;
                var monitoringEvent = EventQueue.Dequeue();
                using (var scope = ServiceProvider.CreateScope())
                {
                    var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                    db.MonitoringEvents.Add(monitoringEvent);
                    db.SaveChanges();
                }

                // TODO Trigger events, notifications
                Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
            }
        });
        _workerThread.IsBackground = true;
        _workerThread.Start();
    }

    private Thread? _workerThread;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        Logger.LogInformation("Monitoring Event Triggering Start");
        _workerThread = new Thread(() =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                if (EventQueue.Count == 0) continue;
                var monitoringEvent = EventQueue.Dequeue();
                using (var scope = ServiceProvider.CreateScope())
                {
                    var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                    db.MonitoringEvents.Add(monitoringEvent);
                    db.SaveChanges();
                }

                // TODO Trigger events, notifications
                Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
            }
        });
        _workerThread.IsBackground = true;
        _workerThread.Start();
    }

That is the code i currently have, and one weird thing is if I debug the code, the service blocks the program execution, while if I use run, it doesn't block and continue as expected. Is there any mistakes there? Also, is there other ways to do this? I tried Task.Run(), Task.Factory.StartNew() and they all block. I'm on Jetbrains Rider on Ubuntu 20.04 if that matters for the threading part. Thanks in advance!

91 Replies

Kouhai•3y ago

There's no way to achieve that, all threads are halted when reaching a breakpoint

kunio_kunOP•3y ago

ah it's not that i put any breakpoints there

kunio_kunOP•3y ago

i meant this buttons

kunio_kunOP•3y ago

If i started it with Debug, it never made it to the next service but it works correctly with Run

kunio_kunOP•3y ago

I have been investigating for a few hours and with Jetbrains' IDE i found that it doesn't get to the TryExecuteBackgroundServiceAsync() and the foreach loop stucks

Kouhai•3y ago

That's weird, do you have specific code that runs for development vs production builds? Does the foreach loop even execute or is _hostedServices empty?

kunio_kunOP•3y ago

it is not empty I have 2 background services and with the debugger breakpoint it didn't evaluate to true for the first background service (this one) also i dont have any specific code that runs on different configuration

Kouhai•3y ago

does hostedService.StartAsync await infinitely? (or it's not even reached)

kunio_kunOP•3y ago

at least not in my own code (non library), and not in this service does it refer to this backgroundservice? it is reached it does have infinite loop I have updated the code to this

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                while (!stoppingToken.IsCancellationRequested)
                {
                    if (EventQueue.Count == 0) continue;
                    var monitoringEvent = EventQueue.Dequeue();
                    using (var scope = ServiceProvider.CreateScope())
                    {
                        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                        await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                        await db.SaveChangesAsync(stoppingToken);
                    }

                    // TODO Trigger events, notifications
                    Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                while (!stoppingToken.IsCancellationRequested)
                {
                    if (EventQueue.Count == 0) continue;
                    var monitoringEvent = EventQueue.Dequeue();
                    using (var scope = ServiceProvider.CreateScope())
                    {
                        var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                        await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                        await db.SaveChangesAsync(stoppingToken);
                    }

                    // TODO Trigger events, notifications
                    Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

ooh

kunio_kunOP•3y ago

kunio_kunOP•3y ago

now wonder if it blocks somewhere else but according to the debugger it doesnt even get to the second backgroundservice i have

Kouhai•3y ago

Based on these logs, I assume both services seem to be started

kunio_kunOP•3y ago

i will try putting breakpoint again

kunio_kunOP•3y ago

kunio_kunOP•3y ago

these are the hostedServices

kunio_kunOP•3y ago

kunio_kunOP•3y ago

kunio_kunOP•3y ago

kunio_kunOP•3y ago

the next step after the last picture is nowhere

kunio_kunOP•3y ago

and the log is also stuck here

kunio_kunOP•3y ago

that was Host.cs, class Host in Microsoft.Extensions.Hosting.Internal but i mean even having breakpoints vs no breakpoints on debug has different result?

Kouhai•3y ago

Hmm so it's stuck after calling StartAsync on the MonitoringEventTriggeringService, I'm not sure if Host.cs has any conditions for production vs development but I doubt that

kunio_kunOP•3y ago

it does get to

if (hostedService is BackgroundService backgroundService)

if (hostedService is BackgroundService backgroundService)

and then next step is nowhere

Kouhai•3y ago

kunio_kunOP•3y ago

okay just tried it again this time the next step was the closing curly brackets of the foreach loop, as if the hostedService is not BackgroundService got to the next enumeration (is it?) in the in _hostedService and then next step is nowhere

Kouhai•3y ago

The condition hostedService is BackgroundService backgroundServic fails? Hmm even if it fails it should continue looping

kunio_kunOP•3y ago

just tried again it didnt continue after the hostedService is BackgroundService part i mean even if it fails, it may be throwing something right?

Kouhai•3y ago

But what would throw? is T just checks if the instance is of that type, if so it would run the code inside the braces

kunio_kunOP•3y ago

while it does inherit from BackgroundService

Kouhai•3y ago

idk if it would help, but what version of ASP.NET Core are you using

kunio_kunOP•3y ago

ASP.NET Core MVC 6

dotnet --version
6.0.402

dotnet --version
6.0.402

Linux 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Linux 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

maybe i should also try it on Windows and see if it happens

Kouhai•3y ago

This SS is for the release build, right thisisalright

kunio_kunOP•3y ago

nope

Kouhai•3y ago

Okay..in that pic it ModBus is run normally

kunio_kunOP•3y ago

if i didnt put breakpoint at all, even with Debug, it reaches that part it does start, but it doesnt run normally like it reaches some point and not continue but if i put some breakpoints and it hit, it doesnt get to even initiate the next service and sometimes it doesnt get to that part of log too (ends on Task is running) on debug

Kouhai•3y ago

Can you do something real quick, wrap all your code that uses Task.Run with try catch, and in catch log any message

kunio_kunOP•3y ago

mmmm wrap in the Task.Run() delegate? so like

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                // start wrapping here?

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                // start wrapping here?

Kouhai•3y ago

Yeah

kunio_kunOP•3y ago

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        if (EventQueue.Count == 0) continue;
                        var monitoringEvent = EventQueue.Dequeue();
                        using (var scope = ServiceProvider.CreateScope())
                        {
                            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                            await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                            await db.SaveChangesAsync(stoppingToken);
                        }

                        // TODO Trigger events, notifications
                        Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                    }
                }
                catch (Exception ex)
                {
                    Logger.LogError(ex, "Something went wrong");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        if (EventQueue.Count == 0) continue;
                        var monitoringEvent = EventQueue.Dequeue();
                        using (var scope = ServiceProvider.CreateScope())
                        {
                            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                            await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                            await db.SaveChangesAsync(stoppingToken);
                        }

                        // TODO Trigger events, notifications
                        Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                    }
                }
                catch (Exception ex)
                {
                    Logger.LogError(ex, "Something went wrong");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

Kouhai•3y ago

Do you have any other service/s that calls Task.Run?

kunio_kunOP•3y ago

i dont I have also commented out other services

kunio_kunOP•3y ago

kunio_kunOP•3y ago

still it didnt make it to the web service

using RemoteMonitor.DatabaseContexts;
using RemoteMonitor.Services;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
    // .AddRazorRuntimeCompilation();
builder.Services.AddDbContext<AppDbContext>();
// builder.Services.AddHostedService<DatabaseMigratorService>();
// builder.Services.AddSignalR().AddMessagePackProtocol();
builder.Services.AddSingleton<MonitoringEventTriggeringService>();
builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());
// builder.Services.AddSingleton<ModbusDeviceWorkerService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<ModbusDeviceWorkerService>());

var app = builder.Build();

if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    app.UseHsts();
}

// app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}"
);

app.Run();

using RemoteMonitor.DatabaseContexts;
using RemoteMonitor.Services;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
    // .AddRazorRuntimeCompilation();
builder.Services.AddDbContext<AppDbContext>();
// builder.Services.AddHostedService<DatabaseMigratorService>();
// builder.Services.AddSignalR().AddMessagePackProtocol();
builder.Services.AddSingleton<MonitoringEventTriggeringService>();
builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());
// builder.Services.AddSingleton<ModbusDeviceWorkerService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<ModbusDeviceWorkerService>());

var app = builder.Build();

if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
    app.UseHsts();
}

// app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}"
);

app.Run();

Hitting the pause button on debug doesnt even pause it Ctrl + C doesnt shut it down

Tvde1•3y ago

You can replace

builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());

builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());

with

builder.Services.AddHostedService<MonitoringEventTriggeringService>();

builder.Services.AddHostedService<MonitoringEventTriggeringService>();

right?

Kouhai•3y ago

Yeah, HostedServices are singletons by default

kunio_kunOP•3y ago

but I wasnt able to resolve it on the other service without registering it as singleton also I now have this instead to try it out:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
    // .AddRazorRuntimeCompilation();
builder.Services.AddDbContext<AppDbContext>();
// builder.Services.AddHostedService<DatabaseMigratorService>();
// builder.Services.AddSignalR().AddMessagePackProtocol();
builder.Services.AddHostedService<MonitoringEventTriggeringService>();
// builder.Services.AddSingleton<MonitoringEventTriggeringService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());
// builder.Services.AddSingleton<ModbusDeviceWorkerService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<ModbusDeviceWorkerService>());

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllersWithViews();
    // .AddRazorRuntimeCompilation();
builder.Services.AddDbContext<AppDbContext>();
// builder.Services.AddHostedService<DatabaseMigratorService>();
// builder.Services.AddSignalR().AddMessagePackProtocol();
builder.Services.AddHostedService<MonitoringEventTriggeringService>();
// builder.Services.AddSingleton<MonitoringEventTriggeringService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<MonitoringEventTriggeringService>());
// builder.Services.AddSingleton<ModbusDeviceWorkerService>();
// builder.Services.AddHostedService(provider => provider.GetRequiredService<ModbusDeviceWorkerService>());

still stuck there

Tvde1•3y ago

that's a lot of commented out code :)

kunio_kunOP•3y ago

yeah

Tvde1•3y ago

but what's the problem? You don't see console output from the task?

kunio_kunOP•3y ago

Nope

Tvde1•3y ago

you might want to put a timer in the while, not to check EventQueue.Count every nano second

kunio_kunOP•3y ago

the app seems seems to be stuck because this service yeah doing this works, like putting a Task.Delay() and await it

Tvde1•3y ago

that works but is not as robust, the PeriodicTimer is really good for these use cases

kunio_kunOP•3y ago

but still it is weird that if I start the app without debugging it, it works

kunio_kunOP•3y ago

run without debugging

Tvde1•3y ago

but you shouldn't have to make a thread or task in your ExecuteAsync

kunio_kunOP•3y ago

meanwhile run with debugging

kunio_kunOP•3y ago

mmmm

Tvde1•3y ago

hmm are you hitting a break point during debugging?

kunio_kunOP•3y ago

i dont but i mean if the host is implemented this way, and the service is background service, then it shouldn't be blocking right?

Tvde1•3y ago

yeah I think it's not blocking it should create a thread for the background service

kunio_kunOP•3y ago

even if i create a thread, with IsBackground set to true it still blocks

Tvde1•3y ago

hmmm

kunio_kunOP•3y ago

but not if i start without debugging still no luck, i tried with break when any exception is thrown also did another try with mute breakpoints didnt see any clue

Kouhai•3y ago

hmmm, can you try adding a log message at the top of Task.Run delegate?

kunio_kunOP•3y ago

sure

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        if (EventQueue.Count == 0)
                        {
                            continue;
                        }

                        var monitoringEvent = EventQueue.Dequeue();
                        using (var scope = ServiceProvider.CreateScope())
                        {
                            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                            await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                            await db.SaveChangesAsync(stoppingToken);
                        }

                        // TODO Trigger events, notifications
                        Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                    }
                }
                catch (Exception ex)
                {
                    Logger.LogError(ex, "Something went wrong");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (LongRunningTask is not null) throw new InvalidOperationException("Already running");

        Logger.LogInformation("Monitoring Event Triggering Start");
        LongRunningTask = Task.Run(async () =>
            {
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        if (EventQueue.Count == 0)
                        {
                            continue;
                        }

                        var monitoringEvent = EventQueue.Dequeue();
                        using (var scope = ServiceProvider.CreateScope())
                        {
                            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                            await db.MonitoringEvents.AddAsync(monitoringEvent, stoppingToken);
                            await db.SaveChangesAsync(stoppingToken);
                        }

                        // TODO Trigger events, notifications
                        Logger.LogInformation($"{monitoringEvent.DateTime} Rule {monitoringEvent.Message} triggered");
                    }
                }
                catch (Exception ex)
                {
                    Logger.LogError(ex, "Something went wrong");
                }
            },
            stoppingToken
        );
        Logger.LogInformation("Task is running");
    }

kunio_kunOP•3y ago

kunio_kunOP•3y ago

also tried with

                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        Logger.LogInformation("Task alive");

                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        Logger.LogInformation("Task alive");

and it's spamming Task alive

Kouhai•3y ago

That's really weird... 😅 I'm honestly lost at what could cause that....

kunio_kunOP•3y ago

using logger to log Task alive blocks but weirdly i have that Console.Write(".") that somehow caused it not to block

kunio_kunOP•3y ago

kunio_kunOP•3y ago

But thanks a lot for at least having a look

Tvde1•3y ago

where did you write the .? in what line of code? but Monitoring Event Triggering Start or Already running is not logged?

kunio_kunOP•3y ago

In the while loop It is, everything is logged, just covered with the dot spam from that service

Tvde1•3y ago

try adding a await Task.Yield(); in the while loop, I'm curious

kunio_kunOP•3y ago

Will try when i'm back to linux Now i'm trying in Windows and i can debug But my Rider version is a few version older Updating it .NET 6 version is also a few version older on my windows machine anyway i also tried this thing before on linux: I started it without debugging, then attach the debugger on the currently running ASP.NET Core 6 app with this same, then attach the debugger And the service stops updated Rider and SDK version on Windows and it still can debug Seems to be linux specific WOW

kunio_kunOP•3y ago

tried this

        LongRunningTask = Task.Run(async () =>
            {
                Task.Yield();
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        Task.Yield();
                        if (EventQueue.Count == 0)
                        {
                            continue;
                        }

        LongRunningTask = Task.Run(async () =>
            {
                Task.Yield();
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        Task.Yield();
                        if (EventQueue.Count == 0)
                        {
                            continue;
                        }

Tvde1•3y ago

await the Task.Yield

kunio_kunOP•3y ago

oh sorry

        LongRunningTask = Task.Run(async () =>
            {
                await Task.Yield();
                Logger.LogInformation("Task Start");
                try

        LongRunningTask = Task.Run(async () =>
            {
                await Task.Yield();
                Logger.LogInformation("Task Start");
                try

seems to be the same but having it in the while loop helps

        LongRunningTask = Task.Run(async () =>
            {
                await Task.Yield();
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        await Task.Yield();

        LongRunningTask = Task.Run(async () =>
            {
                await Task.Yield();
                Logger.LogInformation("Task Start");
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        await Task.Yield();

I'm using await Task.Yield() for now

kunio_kunOP•3y ago

I also opened an issue at Github https://github.com/dotnet/aspnetcore/issues/44706

GitHub

Launch Without Debugging Doesn't Block BackgroundService and Whole ...

Is there an existing issue for this? I have searched the existing issues Describe the bug I have a BackgroundService that looks like this using RemoteMonitor.Classes; using RemoteMonitor.DatabaseCo...

Kouhai•3y ago

It seems like it's a runtime problem Even a very simple code like this

for (int i = 0; i < 3; i++)
{
    Task.Run(() =>
    {
        Console.WriteLine("Task started");
        while (true) { }
    });
    await Task.Delay(500);
}

for (int i = 0; i < 3; i++)
{
    Task.Run(() =>
    {
        Console.WriteLine("Task started");
        while (true) { }
    });
    await Task.Delay(500);
}

Halts after the first Task is started I tested it on WSL running Debian distro

kunio_kunOP•3y ago

I see even without ASP.NET Core 6?

Kouhai•3y ago

Yup, just a very simple console app

kunio_kunOP•3y ago

i see if you don't mind i'll include these as well into the issue

Kouhai•3y ago

Yeah sure, go ahead

kunio_kunOP•3y ago

thanks

Kouhai•3y ago

It was also "fixed" by using await Task.Delay or await Task.Yield, so it seems like infinite loops makes something break on Linux

kunio_kunOP•3y ago

well not so infinite on my previous code, at least checks for stoppingToken

Kouhai•3y ago

True, but practically it was infinite, because stoppingToken doesn't cancel before the whole program halts

kunio_kunOP•3y ago

ah yeah

Gaming

Programming

ASP NET Core 6 Launch without Debugging doesn't block Background Service but Debug Does

Did you find this page helpful?