Sorry, missed this earlier. If it were me, I'd probably solve this on the DB side with `statement_t

Sorry, missed this earlier. If it were me, I'd probably solve this on the DB side with

statement_timeout

statement_timeout

. Cut out all the interactions across the stack and just have queries get kiboshed if they run too long.

https://www.postgresql.org/docs/9.6/runtime-config-client.html

PostgreSQL Documentation

Client Connection Defaults

luiseok•3/11/25, 2:30 PM

Hi! I discovered that Hyperdrive has not been functioning properly since around 04:00 UTC on March 10.

It seems that the connection pool is insufficient, so I increased the pool size in Supabase. However, I am still encountering a "Connection terminated unexpectedly" error. No changes have been made to the code, and I have confirmed that connecting directly to the Supabase database using the connection string from a local environment works without any issues.

Would it be possible to get some assistance regarding this issue?

Brayden•3/11/25, 7:51 PM

Connected two different databases to Hyperdrive and when I try to query against them through a Worker I see it oscillate between the following two errors:

Any known issues happening at the moment or is this more indicative that it's something wrong on my end?

AJROP•3/11/25, 7:52 PM

We're releasing at the moment, but that shouldn't cause either of those. Send me the IDs and I'll take a look?

AJROP•3/11/25, 7:53 PM

Aside - those error messages should not be user-visible. This release includes some improvements to that, but if it's not better after today we'll do another pass on that. Apologies for the lack of visibility there.

kunal•3/11/25, 10:34 PM

I just created a new hyperdrive connection (replacing one regional replica rds instance with another, same database spec but moving regions slightly). I’m seeing much higher/variable latencies than the previous connection (in the latency tab in metrics). I’m wondering if something about my database is slow, or if it takes hyperdrive a while to figure out where to put its pools close to my database?

Kkunal I just created a new hyperdrive connection (replacing one regional replica rds i...

AJROP•3/11/25, 11:50 PM

if you DM me the IDs of new/old I can take a look in the morning

AAJR Sorry, missed this earlier. If it were me, I'd probably solve this on the DB si...

sam•3/12/25, 7:59 PM

huh it actually ended up not being our usage of hyperdrive, nuance in the timing apis, nor our pg statement_timeout

i forgot that we were also utilizing the cf cache api tucked into this codepath, not covered by our wrapper

turns out it was the call that was taking a butt load of time -- so just gotta make sure we wrap all those with timeouts

thanks for the help regardless!

AAJR Aside - those error messages should not be user-visible. This release includes ...

dave•3/13/25, 6:27 AM

I’ve definitely seen them before many times.

Ddave I’ve definitely seen them before many times.

AJROP•3/13/25, 11:05 AM

Okay. Well, if you see them after this week please toss me a heads up. That is not what we want to see.

Nick•3/13/25, 4:34 PM

I'm running into the error on 100% of my requests to a Hyperdrive private database. I think it might be coming from a database error because I've just modified the schemas, but don't see anywhere to view a more specific error message. Is there somewhere where that will be logged?

Nick•3/13/25, 4:56 PM

I may have actually tracked this down to just being a poor network connection between my EC2 instance and the argotunnel endpoints. I ran and that resulted in errors

NNick I may have actually tracked this down to just being a poor network connection be...

dave•3/14/25, 10:58 AM

Which region (and you using IPv6 or IPv4)?Just curious so I don't run into it

Ddave Which region (and you using IPv6 or IPv4)?Just curious so I don't run into it 😛

Nick•3/14/25, 3:07 PM

I got the error message fixed, but the diag command still seems to fail. This VPC is IPv4-only, so maybe that's why it fails? The region is us-west-2b. The other thing worth noting is that it's a dev instance, so I've tried to keep the instance size as small as will function. Ultimately, it's working reliably now by correcting an SQL query that previously had a bad cast.

NNick I'm running into the error `Error: write CONNECTION_CLOSED [some value].hyperdri...

AJROP•3/14/25, 3:33 PM

Mind DM'ing me your Hyperdrive ID so I can dig into why there wasn't more info in that error message?

Chaika•3/14/25, 3:57 PM

Hyperdrive is so cool, the private tunnel connection setup process is very slick and guided, and just works.

I saw someone mention it before, so in case anyone else comes across this: Clickhouse's Postgres interface does kind of work with Hyperdrive, I wouldn't recommend it though.
You have to enable SSL/TLS in Clickhouse config, only use Simple Queries, not extended (so no parameters, and .simple() on

sql

sql

template or .unsafe() for postgres.js). It then works for most queries, but on repeated ones it'll often throw a connection error. The best way to get it to work reliably is to throw an error after your query (like by running ), and Clickhouse Postgres resets the entire connection on all errors, so you'll get a new conn on the next attempt, rip connection pooling though. It works fine for my limited use of small data querying though, and connecting securely through a tunnel.

Clickhouse's MySQL Interface seems to be much more fleshed out and workable, so when Hyperdrive supports that, will probably be much nicer...

vrballer•3/14/25, 11:05 PM

Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's been going great but recently we're hitting scaling issues over 1M requests per day. Specifically around not enough connections.

Using Hyperdrive appeared to be the right choice and the switch seemed relatively simple.

However we've run up against a wall with this error. Queries do seem to work but i guess not all of them? We tried using but no dice

vrballer•3/14/25, 11:13 PM

^ Also not sure if it's related but we do use in some cases

Vvrballer ^ Also not sure if it's related but we do use `jsonb` in some cases

KingMaker•3/14/25, 11:18 PM

Hi, I’m in @VRBaller’s team

Adding onto this, we are doing nested jsonb queries:
for example:

value->>childValue

Vvrballer Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's be...

AJROP•3/15/25, 12:19 AM

We'll go ahead and bump your connection allotment in our next release. Please DM me your Hyperdrive and account IDs.

We'll be planning to cut a release starting early next week.

AAJR We'll go ahead and bump your connection allotment in our next release. Please DM...

AJROP•3/15/25, 12:22 AM

FYI also @KingMaker

Vvrballer Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's be...

AJROP•3/15/25, 12:33 AM

Also, to make sure I'm reading this right, are you having two separate issues? One is scaling, the other is a specific query?

AAJR Also, to make sure I'm reading this right, are you having two separate issues? O...

vrballer•3/15/25, 12:45 AM

We were having connection issues to our Neon Postgres. It seemed like HyperDrive was the right solution for large scale due to it's strong pooling mechanism. That might be working correctly however it seems like fetching data via Hyperdrive is behaving differently

vrballer•3/15/25, 12:46 AM

Essentially we're trying to go from:

to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)

AJROP•3/15/25, 12:48 AM

That is odd. That error is a type error from elsewhere in your code, but ultimately it shouldn't happen if your script had been working before and that's all you changed.

AJROP•3/15/25, 12:49 AM

Honestly I'm not overly familiar with neon's driver or drizzle, so I'm afraid I may not be the best positioned here.

Is the error getting thrown here when you try to connect, or later on when you go to run queries?

AJROP•3/15/25, 12:50 AM

Followup question, are you able to run any queries successfully at all?

vrballer•3/15/25, 12:58 AM

queries are showing up in hyperdrive

vrballer•3/15/25, 12:59 AM

so it's connected correctly

vrballer•3/15/25, 12:59 AM

vrballer•3/15/25, 12:59 AM

You can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)

Vvrballer You can see the queries showed up in Hyperdrive Analytics (while the worker was ...

KingMaker•3/15/25, 7:40 PM

Cc @AJR

KKingMaker Cc @AJR

AJROP•3/15/25, 7:42 PM

If you swap out Hyperdrive for a direct connection with the changed driver do you still see those errors?

Also, nobody's sent me any IDs yet.

kchro3•3/16/25, 12:55 AM

Hi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike in errors, and I don't see any logs in my dashboard so I can't really trace it down. I did see this happen when I was having issues with Hyperdrive, so I wanted to see if I could get a look please!

Kkchro3 Hi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike ...

kchro3•3/16/25, 12:58 AM

around the same time, i see that the number of DatabaseConnections I have in RDS gets really weird, which happened the last time I had a bunch of DB disconnects

Kkchro3 around the same time, i see that the number of DatabaseConnections I have in RDS...

kchro3•3/16/25, 1:29 AM

I'm now fairly convinced that it was a Hyperdrive issue. As a hot-fix, I just deployed with a raw postgresjs client (no hyperdrive) and the errors stopped

an•3/16/25, 1:37 AM

^ I'm experiencing similar issue, not yet deep diving to the trace but the API mostly calling to hyperdrive

about ~1 hour ago everything still good

Aan ^ I'm experiencing similar issue, not yet deep diving to the trace but the API m...

kchro3•3/16/25, 1:38 AM

let me know if you want to share notes on what you find

kchro3•3/16/25, 2:09 AM

this was particularly rough because i didn't see any logs in my worker that would indicate a hyperdrive issue. curious what other people were seeing

ivan•3/16/25, 3:08 AM

Same here, replacing hyperdrive with another pooler solved the issue, was really hard to pinpoint

Iivan Same here, replacing hyperdrive with another pooler solved the issue, was really...

kchro3•3/16/25, 3:16 AM

what did you use?

AJROP•3/16/25, 3:28 AM

Hey all, can you confirm if you're still seeing impact? I'm seeing issues in some of our Virginia locations.

@kchro3 @an @patternmaster

kchro3•3/16/25, 3:29 AM

I don’t know, I disabled Hyperdrive altogether

Kkchro3 what did you use?

ivan•3/16/25, 3:53 AM

I'm using timescaledb and they have their own pooler

AAJR Hey all, can you confirm if you're still seeing impact? I'm seeing issues in so...

ivan•3/16/25, 3:53 AM

I can test tomorrow, my timescaledb is indeed in virginia

AJROP•3/16/25, 4:13 AM

DMing you now to collect some information

AJROP•3/16/25, 5:07 AM

This was escalated as an incident, and has now been remediated. We are monitoring status to ensure things remain functioning as expected.

https://www.cloudflarestatus.com/

Cloudflare Status

Welcome to Cloudflare's home for real-time and historical data on system performance.

onetwoDom•3/16/25, 6:41 AM

I am afraid we were still having issues. I created a new Hyperdrive to the same db and it works fine. CORRECTION: the new hyperdrive worked fine at first, but then Client Disconnected errors started building up again

Sorry, missed this earlier. If it were me, I'd probably solve this on the DB side with `statement_t

Similar Threads