.NET API Memory Leak
Hi guys im seeing a memory leak happening in my .NET API docker application. This is just a straight up generic API template from Visual Studio and no other changes have been done. I've also profiled the application locally but im not seeing the same issue.
Repo link: https://github.com/ayush-lal/railway-.net-api-spike
50 Replies
Project ID:
fb5bd999-4a35-486f-a8a1-326997dae93afb5bd999-4a35-486f-a8a1-326997dae93a
not much we can do for you here except tell you to make sure nixpacks runs your app in the same or as close to your local environment
same .net version, same build parameters, etc etc
is nixpacks being used? Its using a dockerfile
ah well then no nixpacks doesn't come into play at all
but I mean same does apply
its weird because the dockerfile gets auto generated when you create the API template. the only thing i've changed is the paths for it to work properly with railway. Im not getting any memory issues running it locally
you're not getting any memory issues running the app locally, or running the docker image locally
Correct. Im running the profiler again and will attach the stats over the next hour or two
I asked two things, what are you saying correct to lol
you stated two things not asked. Either way im saying correct to both.
i did leave out the question mark but I figured it would have still come across as a question, my bad
Nah all good, was just saying correct to both your questions. One thing i havent tried is using nixpacks over docker, do you think i should try that?
nixpacks over docker?
so remove the dockerfile completely so nixpacks will be used
also, you showed a picture of your project usage, do you see the same stair steps from the service metrics
yeah i do
you can try nixpacks, sure
those metrics span a couple of days, have you ran your app locally for a couple of days without restarts?
it seems like every hour there is a memory increase (1mb) when deployed via Railway. Im not seeing the same when profiling locally. I'll have the profiler running whole day and feedback tonight. I havent run the profiler/api locally for days
Right now i've just deployed another .net 6 api without docker and we'll see how that goes
how do you start your docker image locally
via Jetbrains Rider or Docker CLI/Dashboard
usually rider as its convinent
right, what's the command
docker build -t RailwayTestApi
docker run --rm -it -p 80:80 -p 443:443 RailwayTestApi
alright, does this memory leak on railway happen with any .net app?
im not 100% sure, i'll try deploying a console app
@aleks aren't you a .net dev?
3 services deployed now. I'll feedback later with metrics
sounds good
I'm not sure what's going on looking at the code, I'll look back when you have metrics
this is the repo for the console app and .net 6 api:
https://github.com/ayush-lal/railway-.net-api-spike2
Not really any help but I had a dotnet API in dotnet 7 and it had the exact same issue. Memory leak in prod on railway but not locally. It grows to about 400mb (starts at like 40) and flattens out there. My band-aid solution is to use the App Sleeping which works well because mine is just a side project that rarely is consumed, but if this is a real app that has users not really much help.
@OhLyln thanks for the feedback mate, are you using Docker?
I have a prod .net 7 api and i'd rather not use the app sleeping feature. Hopefully we can get to the bottom of this.
build your image locally, upload it to ghcr and let railway run that image, just to take another variable out of the equation
(they use buildkit and buildx)
great idea thanks. I'll do that tonight
Yep @Ayush I was using my own Dockerfile that looked exactly like yours basically, so no Nix. I did look into it and it seemed like people on GH were hinting at the official MS docker import was causing the leak but I tried with a different version of Dotnet, using 6 and it still persisted, so it's either a leak in both versions or somewhere else in the chain.
thats a great way to say railway is not at fault
We'll see how it goes with the other services using Nix
so even with the api using nixpacks, theres a mem increase every hour (give or take) by 1mb:
console app seems to be stable at 19mb:
docker api is increasing as before. 1mb per hour
well I'll be honest, that's incredibly odd, but I don't really know what to tell you here
Yeah honestly I have no clue either. Luckily in my case App Sleeping works fine (it usually only uses 30mb or so) but if you need it to be up all the time I’m not sure what else would work.
You could try a cron job that restarts the service every day or something, you’d have a minute or two of downtime each day but the memory usage would reset back to normal on each startup.
with a proper healthcheck in place and without a volume, you should have virtually no downtime with a redeploy
thanks guys yeah very weird scenario going on. What do you recommend? Cron to restart the API once a day or app sleeping? Am i understanding correctly that app sleeping essentially stops the container and restarts it as soon as theres traffic?
my honest recommendation, don't use a language that has such issues
It doesn't generally
C# is very good at memory - it's Rust you have to watch out for
Yeah dotnet is usually pretty good with memory but this was a pain for me for like 6 months haha. So App Sleeping will make your app do nothing (and consume no memory) until your api is hit, then it "wakes up". it seems it goes back to sleep after 10 minutes of no consumption. It will save you a lot of money (I literally halved my bill) and is really easy to do, literally just enable it in settings. I'd recommend it, the only downside is the "waking up" of my dotnet app takes about 5-15 seconds, so that first api request will be slow.
thanks @OhLyln, i'll give it a shot. Its just weird that a pretty performant language out of the box is having these issues on Railway but not locally. I havent tried other hosting providers though
Do these work with Railway? (I am not a .net dev, just a humble front end game dev) https://docs.aws.amazon.com/lambda/latest/dg/lambda-csharp.html
Building Lambda functions with C# - AWS Lambda
The following sections explain how common programming patterns and core concepts apply when authoring Lambda function code in C#.
sorry just seeing this now, not really no Railway doesn't have serverless functions (often called lambda bc AWS) but App Sleeping is kind of similar. Serverless functions are bascialy sleeping plus the possibility of scale to infinity