Error: connect ETIMEDOUT/ECONNREFUSED on all db writes

Project ID: 316aa21f-dc55-4e9f-be8f-13129d27d7c0 I'm using the legacy postgres. About 2 hours ago, all db write calls from my application started hanging and resulting in an ETIMEDOUT error. It's now resulting in a ECONNREFUSED error. I have tried redeploying an older version of my application, with no difference. If I build and run the app locally pointed at the prod db, it works normally -- so I believe the issue is with the deployed railway service. current error:
Error: connect ECONNREFUSED 35.227.138.16:6335

at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {

errno: -111,

code: 'ECONNREFUSED',

syscall: 'connect',

address: '35.227.138.16',

port: 6335

}
Error: connect ECONNREFUSED 35.227.138.16:6335

at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {

errno: -111,

code: 'ECONNREFUSED',

syscall: 'connect',

address: '35.227.138.16',

port: 6335

}
Solution:
We just restored service for the Redis.
Jump to solution
9 Replies
Percy
Percy•12mo ago
Project ID: 316aa21f-dc55-4e9f-be8f-13129d27d7c0
angelo
angelo•12mo ago
#🚨|incidents - possibly related, looking into it. So it appears that your production Redis in on Stacker 50. Is the data on Redis requiring persistence, or do you have a depency on that Redis? (I would also migrate your Postgres DB while you can.)
cartermcalister
cartermcalister•12mo ago
redis data does require persistence I will migrate postgres before the cutoff, but can't do it at this moment because of the associated downtime :/
angelo
angelo•12mo ago
No worries here, can you check if the workload is back online?
Solution
angelo
angelo•12mo ago
We just restored service for the Redis.
angelo
angelo•12mo ago
Migration doesn't kill the old service until the new one is healthy.
cartermcalister
cartermcalister•12mo ago
@Angelo yes, it's working again. Thank you for your help!
angelo
angelo•12mo ago
Unfortunately I have been recently informed that we stopped the services while the handover is going on, but I would encourage you to do it sooner vs. later, V2 Databases weren't affected by this outage.
cartermcalister
cartermcalister•12mo ago
noted, we will prioritize it ASAP. ty again for the help
Want results from more Discord servers?
Add your server