Redis down?

Hello, There seems to be connection problems to both Postgres servers and Redis servers (mostly Redis). I get this error all the time and I cannot get the servers to connect: Error: connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 6379 } I tried redeploying, restarting. Did not work. I have not changed anything, it just stopped working. Project ID: 312b74d3-7061-448f-a297-539dd2bfbb7e
28 Replies
Percy
Percy11mo ago
Project ID: 312b74d3-7061-448f-a297-539dd2bfbb7e
thomasmol
thomasmol11mo ago
same for me for a pg database, private db url does not work anymore public one does
dkrusenstrahle
dkrusenstrahle11mo ago
It worked for ten minutes and now it does not work again. Very bad. I have made no code changes that could impact this. Very hard to rely on Railway when these things happens now and then 😦
Patrick
Patrick11mo ago
I'm experiencing something similar where my Postgres and Redis on Digital Ocean can't be reached from Railway services 😬
dkrusenstrahle
dkrusenstrahle11mo ago
Yes it seems that there is an internal networking issue of some sort on Railway because the databases themselfs can be access through for example Postico. @Brody what is happening? It works and then 10 min later it stops working (often after new deploy) @Brody Any ETA when this will be fixed, been many hours now.
Fragly
Fragly11mo ago
#🛂|readme #5
dkrusenstrahle
dkrusenstrahle11mo ago
Well, sorry about the breach of rules but the servers has been down for hours and I get no response and this is serious. My service is unusable.
Fragly
Fragly11mo ago
I completely understand, hopefully someone sees this soon 🙏
Medim
Medim11mo ago
If it's that critical u should consider upgrading to the Team plan. And get email support with the team
dkrusenstrahle
dkrusenstrahle11mo ago
I am on the PRO plan and that includes Email Support I see now. I thought this was the place to get help.
Medim
Medim11mo ago
This is community help A lot of ppl are getting the same issue as you, but i can't tell if it was alerted to the team since seems like no conductors are active rn But it should be adressed soon.
Ray
Ray11mo ago
We're investigating this right now. Apologies for the delay
Ray
Ray11mo ago
We're currently experiencing intermittent networking issues impacti...
We're currently experiencing intermittent networking issues impacting some services in europe-west4 - Incident details - Railway Status
Ray
Ray11mo ago
@dkrusenstrahle
Error: connect ECONNREFUSED 127.0.0.1:6379
Are you sure you're using the correct connection string for Redis? Your issue might be unrelated
dkrusenstrahle
dkrusenstrahle11mo ago
That one may be unrelated perhaps, but the variable is redis_url and that points to the Redis server. But I get other issues. It time out because the connection is not working. No code changed and it has worked up till today and it works for 10-20 min sometimes and then fails again (server crashes after timeout).
Ray
Ray11mo ago
Gotcha And Redis is the one timing out, right?
dkrusenstrahle
dkrusenstrahle11mo ago
From what I can gather from the logs but sometimes postgres also times out. This is the latest logs: at Socket.<anonymous> (/app/node_modules/ioredis/built/Redis.js:170:41) at Object.onceWrapper (node:events:631:28) at Socket.emit (node:events:517:28) at Socket._onTimeout (node:net:598:8) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) { errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' } Error connecting to PostgreSQL: ConnectionAcquireTimeoutError [SequelizeConnectionAcquireTimeoutError]: Operation timeout at ConnectionManager.getConnection (/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:206:15) at async /app/node_modules/sequelize/lib/sequelize.js:305:26 at async Sequelize.authenticate (/app/node_modules/sequelize/lib/sequelize.js:457:5) at async PostgresService.connect (/app/services/postgres_service.js:39:7) { parent: TimeoutError: Operation timeout at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7), original: TimeoutError: Operation timeout at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) } node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); ^ ConnectionAcquireTimeoutError [SequelizeConnectionAcquireTimeoutError]: Operation timeout at ConnectionManager.getConnection (/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:206:15) at async /app/node_modules/sequelize/lib/sequelize.js:305:26 at async Sequelize.authenticate (/app/node_modules/sequelize/lib/sequelize.js:457:5) at async PostgresService.connect (/app/services/postgres_service.js:39:7) { parent: TimeoutError: Operation timeout at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7), original: TimeoutError: Operation timeout at Timeout.<anonymous> (/app/node_modules/sequelize-pool/lib/Deferred.js:17:25) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7) } Node.js v18.18.2 error: script "start" exited with code 1 (SIGHUP) Its like the authentication to the servers are not working. I have changed no code and its been working fine for weeks. I am using the public url if that helps (not internal one)
Ray
Ray11mo ago
Can you paste a link to where which service these logs are from?
dkrusenstrahle
dkrusenstrahle11mo ago
I see them in the backend service logs What link do you need?
Fragly
Fragly11mo ago
I think Ray is trying to figure out which service those logs are from, so a link to your service it would look something like this: railway.app/project/43d250e5-ab9b-4341-8828-5e344d20524b/service/8f1b6722-7196-4cdd-8cb0-27ff50aa7a21
dkrusenstrahle
dkrusenstrahle11mo ago
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
Ray
Ray11mo ago
Are you still seeing this issue?
JustJake
JustJake11mo ago
@dkrusenstrahle @dkrusenstrahle
dkrusenstrahle
dkrusenstrahle11mo ago
Seems to work now What are Railway doing to make sure a thing like this will not happen again. It was very painful and many hours gone to waste. Will there be better safe guards out in place?
angelo
angelo11mo ago
@dkrusenstrahle - What company/Pro workspace are you here on behalf for? I would love to refund you for this month and will share some mitigation steps so that we can try to restore your confidence on the platform.
dkrusenstrahle
dkrusenstrahle11mo ago
Thank you @Angelo , very nice of you! My project id is 312b74d3-7061-448f-a297-539dd2bfbb7e if that helps? BTW, there seems to be some other issues right now. One of my projects has been deploying for 20 minutes now (and counting), it usually takes 3-4 minutes. I really love the simplicity of Railway, you guys really rocks in terms of functionality, support and ease of use. Just get on top of these stability issues and you will be the best. After all, stability is number one all day long.
Fragly
Fragly11mo ago
Your deployment failing to go out is possibly related to: https://status.railway.app/clq29e6po141569bkokxcrhik8b Should be fixed soon 🙂
dkrusenstrahle
dkrusenstrahle11mo ago
Yepp, works now, thanks!
Want results from more Discord servers?
Add your server