Hello, seeing a lot of `Connection terminated unexpectedly` errors suddenly, how can I debug?

Hello, seeing a lot of

Connection terminated unexpectedly

Connection terminated unexpectedly

errors suddenly, how can I debug?

JJeff Hello, seeing a lot of `Connection terminated unexpectedly` errors suddenly, how...

AJR•3/6/25, 4:37 PM

please post or DM your hyperdrive ID.

thomasgauvin•3/7/25, 4:34 PM

We made it official with an announcement

: https://developers.cloudflare.com/changelog/2025-03-04-hyperdrive-pooling-near-database-and-ip-range-egress/

TLDR:

Query latency is reduced by 90% (!) by pooling databases connections closer to your database (as mentioned by @AJR earlier this week)
Hyperdrive uses Cloudflare IP addresses to connect to your database so you can configure access control lists on your database firewall (as mentioned by @xortive earlier this week)
Workers Smart Placement is more effective with Hyperdrive to ensure minimal network latency for db queries

@Hyperdrive ^

Cloudflare Docs

Hyperdrive reduces query latency by up to 90% and now supports IP a...

Tthomasgauvin We made it official with an announcement 📢: https://developers.cloudflare.com/...

Rapid•3/7/25, 4:43 PM

The link in the changelog is

https://developers.cloudflare.com/hyperdrive/platform/changelog/(https://www.cloudflare.com/ips/

https://developers.cloudflare.com/hyperdrive/platform/changelog/(https://www.cloudflare.com/ips/

Rapid•3/7/25, 4:43 PM

https://developers.cloudflare.com/hyperdrive/platform/changelog/#2025-03-07

Rapid•3/7/25, 4:43 PM

Just a malfunctioned link

dan•3/7/25, 6:00 PM

kinda interesting, it seems like my hyperdrive performance has gotten significantly worse in last 24 hours

dan•3/7/25, 6:00 PM

dan•3/7/25, 6:00 PM

connection latency in last 6 hours is 100ms

dan•3/7/25, 6:00 PM

vs 34ms over last week

AJR•3/7/25, 6:34 PM

Mind sending me the ID? We haven't released any changes in the last 24 hours, I'd be curious to see where you're at.

Tthomasgauvin We made it official with an announcement 📢: https://developers.cloudflare.com/...

dave•3/8/25, 12:30 AM

IMO the firewall news so great I wish it was an email! I was very happy to see that news in my RSS feed.

mega•3/8/25, 10:48 AM

has anyone used for pushing data onto a kafka queue?

mega•3/8/25, 10:49 AM

or anything similar (redpanda, celery, some other db etc.)?

Mmega has anyone used for pushing data onto a kafka queue?

dave•3/8/25, 4:07 PM

imo I found traditional queues to be more complicated than needed. I personally recommend just using PostgreSQL and polling a table with a Cron Worker.

mega•3/8/25, 4:21 PM

I agree, but I want to keep some high volume financial data in a backup server (apart from D1), and was wondering if I can push that onto a kafka which later can be processed by something else (aws rds probably).

AJR•3/8/25, 5:40 PM

Folks internally have absolutely used a Cloudflare Queue as part of their architecture, where it eventually lands into an RDBMS. You can't talk to Hyperdrive directly from a queue but you can definitely stick a worker on the end and from there bulk load that data into your DB.

dan•3/9/25, 3:52 PM

I'm curious as to why the docs for postgres.js say max: 5max: 5 instead of max: 1max: 1 for connections? https://developers.cloudflare.com/hyperdrive/configuration/connect-to-postgres/#postgresjs

Cloudflare Docs

Connect to PostgreSQL · Hyperdrive

Hyperdrive supports PostgreSQL and PostgreSQL-compatible databases, popular drivers and Object Relational Mapper (ORM) libraries that use those drivers.

AJR•3/9/25, 3:53 PM

Workers allow a limit of 6 concurrent outgoing connections. If you run into that it can cause some hard-to-debug error scenarios. That's the main concern for that setting, which was my focus when adding that note. If you don't want multiple concurrent queries, max: 1max: 1 is probably better as it's fewer moving parts to debug if you do run into issues.

AAJR Workers allow a limit of 6 concurrent outgoing connections. If you run into that...

AJR•3/9/25, 4:15 PM

There's an open issue to set the default to something safer for Workers, but I'm not sure how that's going to play out: https://github.com/porsager/postgres/issues/1023

GitHub

Default connections to less than 10 in Cloudflare Workers environme...

With the following Cloudflare Worker, errors can be observed when deployed to production: import postgres from 'postgres'; export default { async fetch(request, env, ctx): Promise<Respon...

AAJR Workers allow a limit of 6 concurrent outgoing connections. If you run into that...

dave•3/10/25, 8:22 AM

Is there likely any noticeable difference between 1 and 5?

AAJR Hyperdrive is a transaction mode pooler, so those 25 go a lot further than you m...

dave•3/10/25, 8:25 AM

Interesting, why are you limited to just 60? (Not an issue for me yet in my use-cases.)

I thought Hyperdrive would be able to handle a near infinite number of connections from Workers.

Kkchro3 are there any recommendations on how to update hyperdrive if you want to rotate ...

dave•3/10/25, 8:25 AM

https://www.ai.moda/en/blog/automate-hyperdrive-iam-auth-with-workers

ai.moda

Automate Hyperdrive IAM Authentication with Workers

Learn how to automatically manage rotating credentials and AWS IAM tokens for PostgreSQL databases using Cloudflare Workers and Hyperdrive.

AAJR Plans is a strong word. It's something we've discussed internally, but no firm ...

dave•3/10/25, 8:33 AM

Monthly ping to let the team know I would be willing to pay for this. (Which is saying a lot, since I’m famously a freeloader when it comes to cloud services.)

AAJR Hyperdrive is a transaction mode pooler, so those 25 go a lot further than you m...

dave•3/10/25, 8:56 AM

Oh, I think I have run into this actually.. does this translate to about 60 concurrent Workers then? Or more?

Ddave Oh, I think I have run into this actually.. does this translate to about 60 conc...

AJR•3/10/25, 11:05 AM

Substantially more. We see folks up past 1k QPS with this right now, glancing at logs at the moment.

Ddave Interesting, why are you limited to just 60? (Not an issue for me yet in my use-...

AJR•3/10/25, 11:11 AM

Couple reasons.

Connections do use memory and CPU and those aren't free, even for us.

Given that it's not self service to change it, if we pick a value that's too high Hyperdrive will just knock over free tier offerings from most providers. When we do make this self service we'll allow a higher maximum value to be set.

If folks hit this limit (or expect to) and reach out to us, we can set it higher manually for them for now. Many customers have. It's a good forcing function for us not to put off the self-service work for too long.

At some point we'll probably allow an absurdly high value to be set with some sort of billing implication, such that we can recoup the cost of resources if people really want to blow the doors off something. We've got some internal teams using Hyperdrive for their products set as high as 1500.

Ddave Is there likely any noticeable difference between 1 and 5?

AJR•3/10/25, 11:15 AM

This one is hard for me to answer, as someone who spends more time in C or Rust than in JavaScript.

Each of those really is a separate connection, so if you do actually want to have multiple queries going at the same time than I think probably there is a difference, yes.

But on Workers I think the general design philosophy would be to do more Workers, not more stuff within a Worker, so I'm not sure how often that would come up in practice.

LLaPh Where can I request Hyperdrive for MySQL early access?

AJR•3/10/25, 1:03 PM

@thomasgauvin one for your list. Not sure what info you need.

Beta isn't running yet, but should be pretty soon.

sam•3/10/25, 5:22 PM

anyone know off the top of their head what the best way to timeout a hyperdrive query using postgres.js?

we setup the client with these opts { connect_timeout: 1, max: 5 }{ connect_timeout: 1, max: 5 } and also have some setTimeout(setTimeout( helpers wrapping the whole function issuing the query -- but still see some successful queries record latencies of multiple minutes sometimes

my understanding is that the wall clock in cf workers will only advance when there is external io performed, so setTimeout will often but not always enforce the timeout? (our metrics confirm this, but just making sure i understand why)

maybe just need to add a max_lifetime or idle_timeout to the connection options instead?

(pics are in US-mountain tz, with the cf dashboard showing avg query latency spiking around the same time as we were seeing internal metrics for max latencies higher than all the timeouts we have set in code)

Original message was deleted

AJR•3/10/25, 6:47 PM

Sorry, not sure I'm following.

I would describe the use case of Hyperdrive as: "I, as a user, want to make a fullstack application on Workers, with my database hosted on Postgres somewhere. I will accomplish this by writing a Worker script that uses a standard Javascript Postgres driver/ORM to query my database".

Websockets can certainly play into some architectures depending on your use case, but I'd imagine most of our users do not use them.

TTyveris Regarding the hyperdrive limitations you mentioned, if my plan is to connect to ...

AJR•3/10/25, 7:05 PM

It will be handled automatically. There are two different "types" of connections at play here. The Hyperdrive->Origin connections are limited, and automatically managed. You shouldn't need to do anything to manage those except not leave a transaction open forever (which breaks Postgres anyway so probably don't do that). Whenever a query completes that connection is put back into the pool for other queries to use. The phrase to search for to learn more would be "transaction-mode connection pooler"

The Worker->Hyperdrive connections can be left open, and are not limited. That's what you'd be leaving sitting open with the architecture you're describing. That should work fine. There's a 15-second timeout on those if they're idle, so if you're going to have low traffic you may want to have a "re-open my connection" control flow path, I guess?

AJR•3/10/25, 7:07 PM

The only caveat I'd add is that a websocket connection does count as 1 of the 6 a Worker is allotted, so you'll be down 1 out of the gate and should plan your settings/query paralellism accordingly.

AAJR The only caveat I'd add is that a websocket connection does count as 1 of the 6 ...

Isaac McFadyen•3/10/25, 7:09 PM

Is this also the case for incoming websockets?

Isaac McFadyen•3/10/25, 7:10 PM

Figured that was the case for outgoing concurrent subrequest limit but didn't know it also applied to incoming

AJR•3/10/25, 7:11 PM

Ahhhh. I might be wrong on that. I had always assumed it was both for tcp/websockets. Better to confirm in a Workers channel . Good call, sorry about that.

AAJR Ahhhh. I might be wrong on that. I had always assumed it was both for tcp/websoc...

Isaac McFadyen•3/10/25, 7:28 PM

I assumed it was too, but I'm not sure if it applies to incoming websockets as well as outgoing (which I think the user was referring to? Or at least I read it that way)

Ssam anyone know off the top of their head what the best way to timeout a hyperdrive ...

AJR•3/11/25, 2:52 AM

Sorry, missed this earlier. If it were me, I'd probably solve this on the DB side with statement_timeoutstatement_timeout . Cut out all the interactions across the stack and just have queries get kiboshed if they run too long.

https://www.postgresql.org/docs/9.6/runtime-config-client.html

PostgreSQL Documentation

Client Connection Defaults

luiseok•3/11/25, 2:30 PM

Hi! I discovered that Hyperdrive has not been functioning properly since around 04:00 UTC on March 10.

It seems that the connection pool is insufficient, so I increased the pool size in Supabase. However, I am still encountering a "Connection terminated unexpectedly" error. No changes have been made to the code, and I have confirmed that connecting directly to the Supabase database using the connection string from a local environment works without any issues.

Would it be possible to get some assistance regarding this issue?

Brayden•3/11/25, 7:51 PM

Connected two different databases to Hyperdrive and when I try to query against them through a Worker I see it oscillate between the following two errors:

write CONNECTION_CLOSED 725fa92510fc468cd02c82f3759d365f.hyperdrive.local:5432

write CONNECTION_CLOSED 725fa92510fc468cd02c82f3759d365f.hyperdrive.local:5432

Internal error.

Internal error.

Any known issues happening at the moment or is this more indicative that it's something wrong on my end?

AJR•3/11/25, 7:52 PM

We're releasing at the moment, but that shouldn't cause either of those. Send me the IDs and I'll take a look?

AJR•3/11/25, 7:53 PM

Aside - those error messages should not be user-visible. This release includes some improvements to that, but if it's not better after today we'll do another pass on that. Apologies for the lack of visibility there.

kunal•3/11/25, 10:34 PM

I just created a new hyperdrive connection (replacing one regional replica rds instance with another, same database spec but moving regions slightly). I’m seeing much higher/variable latencies than the previous connection (in the latency tab in metrics). I’m wondering if something about my database is slow, or if it takes hyperdrive a while to figure out where to put its pools close to my database?

Kkunal I just created a new hyperdrive connection (replacing one regional replica rds i...

AJR•3/11/25, 11:50 PM

if you DM me the IDs of new/old I can take a look in the morning

AAJR Sorry, missed this earlier. If it were me, I'd probably solve this on the DB si...

sam•3/12/25, 7:59 PM

huh it actually ended up not being our usage of hyperdrive, nuance in the timing apis, nor our pg statement_timeout

i forgot that we were also utilizing the cf cache api tucked into this codepath, not covered by our setTimeout(setTimeout( wrapper

turns out it was the cache.get(cache.get( call that was taking a butt load of time -- so just gotta make sure we wrap all those with timeouts

thanks for the help regardless!

AAJR Aside - those error messages should not be user-visible. This release includes ...

dave•3/13/25, 6:27 AM

I’ve definitely seen them before many times.

Ddave I’ve definitely seen them before many times.

AJR•3/13/25, 11:05 AM

Okay. Well, if you see them after this week please toss me a heads up. That is not what we want to see.

Nick•3/13/25, 4:34 PM

I'm running into the error Error: write CONNECTION_CLOSED [some value].hyperdrive.local:5432Error: write CONNECTION_CLOSED [some value].hyperdrive.local:5432 on 100% of my requests to a Hyperdrive private database. I think it might be coming from a database error because I've just modified the schemas, but don't see anywhere to view a more specific error message. Is there somewhere where that will be logged?

Hello, seeing a lot of `Connection terminated unexpectedly` errors suddenly, how can I debug?

Similar Threads

Similar Threads

Similar Threads