App periodically loses connection to postgres
Every few days my web app running in production loses access to the postgres DB I have also deployed via Railway. I see a "connection timeout" error from the client logs, and nothing gets logged to the server according to the Railway observability tab. The only way I have been able to fix it is triggering a re-deployment which presumably rolls the Kubernetes pods. Any ideas?
28 Replies
Project ID:
80d4d55f-409c-4c75-9ab3-61d3cefe2e4680d4d55f-409c-4c75-9ab3-61d3cefe2e46
What kind of app are you running? Are you using private networking?
It's a bun app running JS on server and client. Using the postgres library (https://github.com/porsager/postgres) to connect to the DB. Not doing anything special with networking that I know of
Okay, bummer. Unfortunately I can't provide much help, but while you wait on people that can help, could you post the error you're receiving? Those are usually very valuable pieces of info :)
I've noticed the same thing.
dan, are you using a pooled connection?
@jtourkos - if you arent using bun or the postgres library, please open your own help thread
Sure, yes 👍
(I am not using bun. My app's a nodejs. Posted here cause I also use a postgres on Railway and didn't want to create dublicate. I'll open a new one if the issue remains or the solution to this one does cover my case as well).
are you using the postgres library in code?
not quite asking about the database you have deployed
no
what kind of app do you have
nodejs with pg and sequelize
But sorry, I didn't mean to highjack the issue here 😊
And I cannot confirm that I keep having this issue till the logs are back.
are you using a pool?
yeah the logging issues really hampers debugging
Not explicitly. I use the default sequelize options.
show the error please
I don’t have logs 😊
haha yeah but I'm sure there's logs from a previous (but same) error right?
Need to check but I left office now. Let’s see how the logging resolution goes and when I’ll go back to my machine I’ll search and post here if I find anything
@Brody Sorry just seeing this. Yes I believe I am using pooled connections, which is the default behavior in the postgres library. Do you think I need to set a timeout option like
idle_timeout here? https://github.com/porsager/postgres#connection-timeoutcan you set the pool minimum to 0 and see how that goes?
pool minimum? Do you mean max number of connections?
i mean the minimum number of connections, aka pool minimum, set it to zero
hmm I don't see that anywhere in the options interface
not being able to set the minimum to zero causes issues when running in a docker environment
I'm gonna try setting
idle_timeout, I wonder if the max connections (default 10) is getting reached and the railway pods are running out of memoryrailway does not use k8s
the postgres databases have 8gb of ram available, same as your plan allows for
Thanks for the help!
but yeah this is just a config issue, not so much an issue with railway, not closing the connections properly, leaving stale connections open without marking them as closed, etc etc