R
Railway•8mo ago
Prodigga

RAILWAY_DEPLOYMENT_DRAINING_SECONDS and debugging graceful shutdowns

Hi there, I saw that I could set RAILWAY_DEPLOYMENT_DRAINING_SECONDS to give my change an app to shut down gracefully. I have some logic in there that works fine on Heroku - I can see my dynos cleaning up and shutting down when I scale down my dynos. I see all my clean up logs and messages. When I try the same on Railway, I don't see any logs. I am not sure if my app is gracefully shutting down. I am not sure how to debug this.
Solution:
the fix was not to use npm run ... and use node ... instead
Jump to solution
49 Replies
Percy
Percy•8mo ago
Project ID: N/A
Brody
Brody•8mo ago
are you catching sigterm?
Prodigga
Prodigga•8mo ago
yes, should be, i use enableShutdownHooks in my nest app which by default catches...
SIGHUP = 'SIGHUP',
SIGINT = 'SIGINT',
SIGQUIT = 'SIGQUIT',
SIGILL = 'SIGILL',
SIGTRAP = 'SIGTRAP',
SIGABRT = 'SIGABRT',
SIGBUS = 'SIGBUS',
SIGFPE = 'SIGFPE',
SIGSEGV = 'SIGSEGV',
SIGUSR2 = 'SIGUSR2',
SIGTERM = 'SIGTERM',
SIGHUP = 'SIGHUP',
SIGINT = 'SIGINT',
SIGQUIT = 'SIGQUIT',
SIGILL = 'SIGILL',
SIGTRAP = 'SIGTRAP',
SIGABRT = 'SIGABRT',
SIGBUS = 'SIGBUS',
SIGFPE = 'SIGFPE',
SIGSEGV = 'SIGSEGV',
SIGUSR2 = 'SIGUSR2',
SIGTERM = 'SIGTERM',
Brody
Brody•8mo ago
do you see logs during regular app activity?
Prodigga
Prodigga•8mo ago
yeah I see my app working normally, and when I restart or redeploy my service it just 'cuts out' where it is and then the new one boots up
Brody
Brody•8mo ago
okay thanks for the info, ill try to get a proof of concept working
Prodigga
Prodigga•8mo ago
ok that would be fantastic, thanks. I thought maybe it is working in the background but the logs aren't being captured anymore? But i have no way of verifying it without seeing the logs
Brody
Brody•8mo ago
one idea popped into my head, what logger are you using?
Prodigga
Prodigga•8mo ago
just the default one what nest comes with
Brody
Brody•8mo ago
have you disabled log buffering?
Prodigga
Prodigga•8mo ago
just checking, sorry i dont specify, and I can see that the default for bufferLogs is false
Brody
Brody•8mo ago
sounds good, i shall get back to my tests
Prodigga
Prodigga•8mo ago
:salute: i will let you know if i discover anything, i am poking around too i wonder if it is because i am using newrelic and starting my app that way, ie node -r newrelic dist/apps/ssr-game-server/main maybe the sigterm is not being propogated properly
Brody
Brody•8mo ago
that is very possible i wouldn't know if newrelic forwards the signals along, are you doing the same on heroku?
Prodigga
Prodigga•8mo ago
no, so it very well might be the culprit
Brody
Brody•8mo ago
try without it
Prodigga
Prodigga•8mo ago
roger
Brody
Brody•8mo ago
it does work, i set it to 15, the default being 3
Brody
Brody•8mo ago
any luck taking newrelic out of the equation?
Prodigga
Prodigga•8mo ago
no luck here, hmm
Brody
Brody•8mo ago
nixpacks or dockerfile?
Prodigga
Prodigga•8mo ago
nixpacks
Brody
Brody•8mo ago
iirc nixpacks wraps the start command in a shell, that also might not be passing down the signals
Prodigga
Prodigga•8mo ago
hmm...
Brody
Brody•8mo ago
how hard would it be to write a dockerfile for your app? it would give you much better control and allow you to ensure the node process is started in exec mode
Prodigga
Prodigga•8mo ago
I have no experience in it, so honestly I have no idea! I am transitioning from Heroku to Railway, solo indie game dev wearing so many hats, haha. My experience is limited. Though I am not doing anything fancy. It's just a boring old nodejs/nest app end of day now.. tomorrow problem... thanks for all the help thus far
Brody
Brody•8mo ago
no problem, ill see if i can replicate this issue when starting my app wrapped in a shell like how nixpacks starts node apps there is an issue on nixpacks about this https://github.com/railwayapp/nixpacks/issues/1032 but neither me or dush can reproduce, and i've tried wrapping my app in a bash and sh shell. i think it might still be worth a try to write a dockerfile, i can try to help you with that if you'd like?
Prodigga
Prodigga•8mo ago
ok, back in the office...! i'd love a hand with that, sounds like a good test considering the open issue
Brody
Brody•8mo ago
for me to have a good chance at writing a working dockerfile for you, please share your most recent build log, anything you have changed nixpacks or service settings wise, and your repo
Prodigga
Prodigga•8mo ago
Hey thanks for offering, i did some more digging and I belive i've found the root cause
Brody
Brody•8mo ago
oh yeah? do tell!
Prodigga
Prodigga•8mo ago
but having a hard time verifying as there seems to be some issues with logs right now..? I have a scattering of logs missing from obervability
Brody
Brody•8mo ago
hmmm I haven't seen any other reports of that, what kind of logs are you missing?
Prodigga
Prodigga•8mo ago
my deploy logs for a fresh deploy look like this: - nothing - business as usual It's missing the typical 'Nest app started' etc. Its not even visible in the Observability panel when i narrow down to the specific service ...
No description
Brody
Brody•8mo ago
incorrect log level? are we sure next doesn't buffer logs by default?
Prodigga
Prodigga•8mo ago
my typical logs look like for the same service
No description
Prodigga
Prodigga•8mo ago
mmm pretty certain, and I haven't touched that anyway, lets ignore that for now, I did observe a log that shows my shutdown signals are working
Brody
Brody•8mo ago
what was the fix?
Prodigga
Prodigga•8mo ago
No description
Prodigga
Prodigga•8mo ago
though, i mentioned the log thing because this is just a few of my graceful shutdown logs haha (I have some logs missing within the same function that those 2 are from hah)
Solution
Prodigga
Prodigga•8mo ago
the fix was not to use npm run ... and use node ... instead
Prodigga
Prodigga•8mo ago
https://help.heroku.com/ROG3H81R/why-does-sigterm-handling-not-work-correctly-in-nodejs-with-npm so for me it was a matter of changing my custom start command from npm run start-game-server:newrelic to node -r newrelic dist/apps/ssr-game-server/main which is just what the start-game-server:newrelic command is, anyway
Brody
Brody•8mo ago
I thought you where using node -r already?
Prodigga
Prodigga•8mo ago
no, i had a custom start command as i have a mono repo and had to specify which to run
Brody
Brody•8mo ago
ah I see
Prodigga
Prodigga•8mo ago
anyway thanks for helping me trouble shoot 🙂
Brody
Brody•8mo ago
happy to help where I can
Brody
Brody•8mo ago
if you are still having issues with logs, please open another thread! 🙂
Want results from more Discord servers?
Add your server