Some services are not responding

Some of my services are not responding to requests. Im getting timeouts.
Solution:
I got it to work, turns out railway changed the database private url from postgres-a1234.railway.internal:5432 to postgres.railway.internal:5432 . This caused any DB related code to timeout the worked, which also caused the /api/ping/ route to timeout till the worker restarted. This was the only difference in the replica, which already had the non-suffixed URl version for its own DB. Not sure what to make of this, and whether its the app's fault. We can close this thread now.
Jump to solution
35 Replies
Percy
Percy9mo ago
Project ID: f433fcca-164a-41f2-ba64-3fee6f309a1b
Augustus
Augustus9mo ago
f433fcca-164a-41f2-ba64-3fee6f309a1b
Brody
Brody9mo ago
could you send some examples service domains?
Augustus
Augustus9mo ago
Will that be published publicly using AnswerOverflow?
Brody
Brody9mo ago
yes you can always delete it afterwards (and that would delete it from answer overflow as well)
Augustus
Augustus9mo ago
Ok. Use any credentials and it just times out The signin page is cached
Brody
Brody9mo ago
is this consistently timing out for you?
Augustus
Augustus9mo ago
Yes for the last hour THe exact replica is hosted on a staging server and that is working perfectly I tried redeploying, restarting, everything Just this project is misbehaving
Brody
Brody9mo ago
what is the tech stack of this app? may I ask why you believe this to be an outage, and not your app misbehaving?
Augustus
Augustus9mo ago
Because even simple endponits such as /api/ping/ that does nothing but reply a hardcoded hello JSON isnt working consistently It works once in 10 to 15 min Rest of the time its timing out
Brody
Brody9mo ago
that's unfortunately not definitive proof, so let's keep an open mind going forward
Augustus
Augustus9mo ago
Sure Also I have the exact replica in another project thats working fine
Brody
Brody9mo ago
the log in is just infinite loading for me, but I'm on mobile right now, will take a closer look when at the computer
Augustus
Augustus9mo ago
There are no logs showing up in this service as well for me to debug anything
Brody
Brody9mo ago
how long ago was this service deployed?
Augustus
Augustus9mo ago
Occassionally it just says workertimeout And it restarts 2 hours ago last First deployed more than a year ago
Brody
Brody9mo ago
that's pointing more towards an issue with your app than an issue with railway
Augustus
Augustus9mo ago
Working perfectly till today It worked perfectly for a year
Brody
Brody9mo ago
does your app contact any 3rd party apis?
Augustus
Augustus9mo ago
Not the /api/ping/ route
Brody
Brody9mo ago
do the metrics look normal?
Augustus
Augustus9mo ago
Yes
Brody
Brody9mo ago
what's your start command
Augustus
Augustus9mo ago
I didnt change anything related to the settings as well It was working just fine yesterday too
Brody
Brody9mo ago
have you tried re-deploying?
Augustus
Augustus9mo ago
Yes Redeployed all services in this project
Brody
Brody9mo ago
no one else has reported anything wrong, so at this time i believe this is an issue with your app
Brody
Brody9mo ago
im seeing this page during a login attempt, meaning railway is working, but your app isnt
No description
Augustus
Augustus9mo ago
Alright, I have a feeling it's an isolated incident. Something similar to this happens ages ago. Had an email thread with Angelo. He resolved it just for our service. Rest is up to the railway guys. I'll move this service away shortly since a customer is very frustrated.
Brody
Brody9mo ago
you said your service was logging a worker timeout, indicating your app is timing out something in your code is blocking all requests from completing and timing out
Augustus
Augustus9mo ago
Hmm let me check
Brody
Brody9mo ago
i understand how this can look like an outage from your perspective, and i totally get all of your reasoning behind thinking that, but i've been doing community support for a long time and ive seen far stranger issues that looked even more like an issue with railway but turned out to be code issues, so please spend some time combing over all of your code and add some verbose debug logging to help you find out where your code is freezing up at
Augustus
Augustus9mo ago
Makes sense. I'll keep an open mind.
Solution
Augustus
Augustus9mo ago
I got it to work, turns out railway changed the database private url from postgres-a1234.railway.internal:5432 to postgres.railway.internal:5432 . This caused any DB related code to timeout the worked, which also caused the /api/ping/ route to timeout till the worker restarted. This was the only difference in the replica, which already had the non-suffixed URl version for its own DB. Not sure what to make of this, and whether its the app's fault. We can close this thread now.
Brody
Brody9mo ago
definitely not the apps fault, but railway isn't just going to change names like that either, though that's for sure what caused it
Want results from more Discord servers?
Add your server