Our n8n instances have possibly started crashing with the PG bouncer / new database setup
Woke up this morning and none of them were responding, a restart to n8n seems to have fixed the connection issue, something is not quite right.
18 Replies
Project ID:
34cc6607-4bf8-44c2-bed8-b7e4fbbe338634cc6607-4bf8-44c2-bed8-b7e4fbbe3386
Other proxies are working good, looks like it is specifically the n8n attached ones that are causing troubles.
Wondering if it's safer to connect directly to the v2 database instance or not.
while I don't know why this is happening exactly, that was my thought process, use the proxy during transition and then after all is said and done, swap back to the v2's database url directly.
I asked to keep the bouncer instances in front of it because I assumed that would generally be more stable. I guess that was a bad idea.
I have support tickets open from this, and this service lets people give us money so if someone could confirm if it's safer to move over to a direct db connection, that would be good to know.
I can keep restarting the instance when issues happen for the short term solution.
what version of n8n are you running?
The important one is on: 1.22.6
It seems like my other n8n, which is a few versions behind that one, is not throwing the same errors in the logs, but has similar errors on the bouncer.
thats the latest, jack's n8n template would deploy that version too, and his template uses the private network to connect to postgres just fine, so i cant see why it wouldnt work just fine.
n8n doesnt appear to use a url formatted variable, so you would need to raw edit the variables, if you want to proceed i can get you the the variables you would need to change.
you can also revert back to the version that isnt throwing errors?
these might be normal on the bouncer, every one of the bouncer instances is logging something similar.
idk why they would use an error log level for that though
that is super common, postgres itself will log to stderr for nearly everything too
when you see these, does it break anything?
Not exactly, as far as I can tell (Hard to debug directly with production traffic). It appears to be staying up since restarting. I assumed it crashed because it opened too many connections just based on all of the logging data.
tbh that does seem like an n8n issue, but pgbouncer is not a fully transparent proxy so it could be causing those warnings, so yeah let me know what you wanna do, swap to the direct postgres variables or rollback to an older version of n8n
or just leave it how it is since they seem like just warnings and info logs
I'll keep pinging it, and update the thread if I see it crash again. It's never done this with the old db so that is the variable that changed....
I'm going to hold on this for a bit so I can observe it.
sounds good!
This is also part of the logging data, just posting it here in case anyone is looking into it.
Seems to be stable, hasn't crashed again. Not sure what happened here.
good to hear!