Neon•2y ago

correct-apricot

Unstable connection with psql

The psql ssl connection drops all the time and then it's not possible to reconnect:

psql: error: connection to server at "pg.neon.tech" (3.136.248.123), port 5432 failed: Operation timed out
    Is the server running on that host and accepting TCP/IP connections?

psql: error: connection to server at "pg.neon.tech" (3.136.248.123), port 5432 failed: Operation timed out
    Is the server running on that host and accepting TCP/IP connections?

This is becoming really frustrating as the only way to continue working on the db is using the browser console (which always works). Anybody experience the same issue? Any ideas how to solve this?

27 Replies

fair-rose•2y ago

I would suggest using the proper connection string, as shown in the console. it would look something like psql 'postgresql://user:password@endpoint.region.aws.neon.tech/database?sslmode=require'

correct-apricotOP•2y ago

thanks for the recommendation but that has the same issues for me (connection timing out then not being able to reconnect for ~5 mins)

dependent-tan•2y ago

cc @martin. Martin and I were also having the same issues Conrad, I wonder if there is some issue with the proxy

harsh-harlequin•2y ago

I tested it on a pretty much clean computer setup (Intel Mac) but still the same issue. I'm also using the built-in database tool in WebStorm now using the full connection string and no issues there

fair-rose•2y ago

What regions are you in? I'll see if there's anything I can find on our end

harsh-harlequin•2y ago

I'm in eu-central-1

fair-rose•2y ago

I can't say I'm experiencing any issues myself, and I don't see any noticable trends in our metrics implying that this is wide spread. @martin and @henrik can you send me your endpoint ID so I can take a look specifically at your endpoints?

fair-rose•2y ago

There does seem to be something wide-spread going on: https://edition.cnn.com/2024/03/04/business/red-sea-cables-cut-internet/index.html https://www.bbc.co.uk/news/technology-68483732 https://downdetector.com/status/google/ just grasping at straws tbh

CNN

Red Sea cables have been damaged, disrupting internet traffic | CNN...

Damage to submarine cables in the Red Sea is disrupting telecommunications networks and forcing providers to reroute as much as a quarter of traffic between Asia, Europe and the Middle East, including internet traffic.

BBC News

Facebook and Instagram in apparent global outage

Meta's platforms are experiencing issues worldwide, according to tracking website Downdetector.

correct-apricotOP•2y ago

I'm in eu-central-1 as well The issue has been going on for weeks, nothing fresh

fair-rose•2y ago

Oh, well that's more interesting. Can you describe the symptoms exactly? I don't fully understand

correct-apricotOP•2y ago

Sent you the project id via dm The issue arises when using psql connections to neon exclusively. Passwordless or normal auth doesn't matter. When the connection is idle for some time and a new query is entered, it says ssh connection has timed out, attempting reconnect. This usually doesn't work (sometimes it does after minutes) and the terminal is stuck in this mode. Even when closing the terminal and opening a fresh one and trying to connect, the connection cant be established and psql will just be stuck

fair-rose•2y ago

Hmm, ok. I'll try and reproduce that

correct-apricotOP•2y ago

I will try to reproduce with screenshots Having to re establish the connection after some idle time make sense, but being stuck there and and not being able to open another one is inconvenient

fair-rose•2y ago

Does this happen every time your endpoint goes idle?

correct-apricotOP•2y ago

Not every time but most times

dependent-tan•2y ago

@Conrad Ludgate can confirm that just letting the connection sit for like 10 minutes forces a reconnection This happened weeks ago, so shouldn't be related to current events

fair-rose•2y ago

Having it reconnect is expected behaviour, but I've never experienced it being stuck before this is what I see:

main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=>

main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=> select 1;
WARNING:  terminating connection due to immediate shutdown command
SSL connection has been closed unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
psql (16.1, server 15.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
main=>

But it reconnects immediately

dependent-tan•2y ago

It's already been too long since I talked with Martin about this. Already paged out of memory 😦 it would be nice to make the passwordless auth, not have to go through the whole bbrowser ux again

correct-apricotOP•2y ago

It gets stuck at Attempting reset: when the issue happens I had no luck reproducing yet 😄 will keep you posted

harsh-harlequin•2y ago

Yeah I kind of forgot it as well after I started to use the built in SQL browser in WebStorm - but every then when I'm in the terminal and it times out it becomes somewhat disruptive to have to go through the browser again. I think for me it times out sooner than 10 mins

fair-rose•2y ago

Yes, that goes for my first suggestion of using the proper connection strings, and not psql -h pg.neon.tech

correct-apricotOP•2y ago

it got hung up again, but this time without the "lost connection. attempting reconnect"

correct-apricotOP•2y ago

the connection was established using the full connection string another bug that i noticed is that when connecting auth, \c to change the db doesn't work. it prompts opening the browser for reauthenticating but stays stuck on the previous db

dependent-tan•2y ago

@henrik if you could give step by step directions, I would like to take a look to see if I can reproduce.

correct-apricotOP•2y ago

Unfortunately I really can't give step by step reproduction directions. The problem seems to be flaky. It's just that sometimes the connection times out and fails to reconnect. I have even noticed it in the Neon console as well, where sometimes a query will take forever (minutes) then fail. Seems to be related to the region. Overall, neon seems to work quite well most of the time, but sometimes you just cannot work at the db because your queries don't come through.

other-emerald•2y ago

I think I might've a similar problem, while migrating from a managed Postges solution. I have a serverless python function connecting to Neon, and it runs perfectly fine at first, but after some inactivity (5 min) the connection is no longer alive. This is as expected (cluster scales to zero after 5 min) - see https://neon.tech/docs/connect/connection-latency#check-the-status-of-a-compute Could that be an explanation for you as well? For me it gave either some EOF error, or hang until a function timeout. Not I'm trying to set the code up to somehow keep the connection alive with "keepalive" messages or to check if a connection is dead. No luck yet. https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-KEEPALIVES https://stackoverflow.com/questions/26741175/psycopg2-db-connection-hangs-on-lost-network-connection https://stackoverflow.com/questions/24130305/postgres-ssl-syscall-error-eof-detected-with-python-and-psycopg Would love to hear suggestions on how to check (probably will have to pre-check with good old "SELECT 1;" statement).

Stack Overflow

Psycopg2 db connection hangs on lost network connection

Problem description I'm using psycopg2 to connect to my PostgreSQL database on a remote host. I open a connection and wait for requests, then for each request I run queries on the connection and r...

Stack Overflow

Postgres SSL SYSCALL error: EOF detected with python and psycopg

Using psycopg2 package with python 2.7 I keep getting the titled error: psycopg2.DatabaseError: SSL SYSCALL error: EOF detected It only occurs when I add a WHERE column LIKE ''%X%'' clause to my

Neon

Connection latency and timeouts - Neon Docs

Neon's Autosuspend feature ('scale to zero') is designed to minimize costs by automatically scaling a compute resource down to zero after a period of inactivity. By default, Neon scales a compute to...

PostgreSQL Documentation

34.1. Database Connection Control Functions

34.1. Database Connection Control Functions # 34.1.1. Connection Strings 34.1.2. Parameter Key Words The following functions deal with making a connection …

correct-apricotOP•2y ago

I don't think you can achieve a permanent connection w/ a serverless db. In my case the issue was when working with the db manually via psql. Unfortunately at times even in the neon console. I haven't had issues with my connected apps yet – but so far I have been mostly dabbling. I am considering using neon for a bigger project. But when you can't access your db for manual migrations / queries for several minutes it's really frustrating. Hope this gets resolved. As soon as I am able to I will keep everyone posted w/ possible reproduction steps

Gaming

Programming

Unstable connection with psql

Did you find this page helpful?