Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's been going great but r

Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's been going great but recently we're hitting scaling issues over 1M requests per day. Specifically around not enough connections.

Using Hyperdrive appeared to be the right choice and the switch seemed relatively simple.

However we've run up against a wall with this error. Queries do seem to work but i guess not all of them? We tried using

fetch_types

fetch_types

but no dice

vrballerOP•3/14/25, 11:13 PM

^ Also not sure if it's related but we do use

jsonb

jsonb

in some cases

Vvrballer ^ Also not sure if it's related but we do use `jsonb` in some cases

KingMaker•3/14/25, 11:18 PM

Hi, I’m in @VRBaller’s team

Adding onto this, we are doing nested jsonb queries:
for example:

value->>childValue

Vvrballer Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's be...

AJR•3/15/25, 12:19 AM

We'll go ahead and bump your connection allotment in our next release. Please DM me your Hyperdrive and account IDs.

We'll be planning to cut a release starting early next week.

AAJR We'll go ahead and bump your connection allotment in our next release. Please DM...

AJR•3/15/25, 12:22 AM

FYI also @KingMaker

Vvrballer Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's be...

AJR•3/15/25, 12:33 AM

Also, to make sure I'm reading this right, are you having two separate issues? One is scaling, the other is a specific query?

AAJR Also, to make sure I'm reading this right, are you having two separate issues? O...

vrballerOP•3/15/25, 12:45 AM

We were having connection issues to our Neon Postgres. It seemed like HyperDrive was the right solution for large scale due to it's strong pooling mechanism. That might be working correctly however it seems like fetching data via Hyperdrive is behaving differently

vrballerOP•3/15/25, 12:46 AM

Essentially we're trying to go from:

export function createDbConnection(databaseUrl: string) {
  const sql = neon(databaseUrl);
  return drizzle(sql, { schema });
}

export function createDbConnection(databaseUrl: string) {
  const sql = neon(databaseUrl);
  return drizzle(sql, { schema });
}

to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)

export function createDbConnection(hyperDrivedatabaseUrl: string) {
  client = postgres(hyperDrivedatabaseUrl,{
    max: 1,
    fetch_types: false,
  },);

  return drizzle(client, { schema });
}

export function createDbConnection(hyperDrivedatabaseUrl: string) {
  client = postgres(hyperDrivedatabaseUrl,{
    max: 1,
    fetch_types: false,
  },);

  return drizzle(client, { schema });
}

AJR•3/15/25, 12:48 AM

That is odd. That error is a type error from elsewhere in your code, but ultimately it shouldn't happen if your script had been working before and that's all you changed.

AJR•3/15/25, 12:49 AM

Honestly I'm not overly familiar with neon's driver or drizzle, so I'm afraid I may not be the best positioned here.

Is the error getting thrown here when you try to connect, or later on when you go to run queries?

AJR•3/15/25, 12:50 AM

Followup question, are you able to run any queries successfully at all?

vrballerOP•3/15/25, 12:58 AM

queries are showing up in hyperdrive

vrballerOP•3/15/25, 12:59 AM

so it's connected correctly

vrballerOP•3/15/25, 12:59 AM

vrballerOP•3/15/25, 12:59 AM

You can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)

Vvrballer You can see the queries showed up in Hyperdrive Analytics (while the worker was ...

KingMaker•3/15/25, 7:40 PM

Cc @AJR

KKingMaker Cc @AJR

AJR•3/15/25, 7:42 PM

If you swap out Hyperdrive for a direct connection with the changed driver do you still see those errors?

Also, nobody's sent me any IDs yet.

kchro3•3/16/25, 12:55 AM

Hi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike in errors, and I don't see any logs in my dashboard so I can't really trace it down. I did see this happen when I was having issues with Hyperdrive, so I wanted to see if I could get a look please!

Kkchro3 Hi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike ...

kchro3•3/16/25, 12:58 AM

around the same time, i see that the number of DatabaseConnections I have in RDS gets really weird, which happened the last time I had a bunch of DB disconnects

Kkchro3 around the same time, i see that the number of DatabaseConnections I have in RDS...

kchro3•3/16/25, 1:29 AM

I'm now fairly convinced that it was a Hyperdrive issue. As a hot-fix, I just deployed with a raw postgresjs client (no hyperdrive) and the errors stopped

an•3/16/25, 1:37 AM

^ I'm experiencing similar issue, not yet deep diving to the trace but the API mostly calling to hyperdrive

about ~1 hour ago everything still good

Aan ^ I'm experiencing similar issue, not yet deep diving to the trace but the API m...

kchro3•3/16/25, 1:38 AM

let me know if you want to share notes on what you find

kchro3•3/16/25, 2:09 AM

this was particularly rough because i didn't see any logs in my worker that would indicate a hyperdrive issue. curious what other people were seeing

ivan•3/16/25, 3:08 AM

Same here, replacing hyperdrive with another pooler solved the issue, was really hard to pinpoint

Iivan Same here, replacing hyperdrive with another pooler solved the issue, was really...

kchro3•3/16/25, 3:16 AM

what did you use?

AJR•3/16/25, 3:28 AM

Hey all, can you confirm if you're still seeing impact? I'm seeing issues in some of our Virginia locations.

@kchro3 @an @patternmaster

kchro3•3/16/25, 3:29 AM

I don’t know, I disabled Hyperdrive altogether

Kkchro3 what did you use?

ivan•3/16/25, 3:53 AM

I'm using timescaledb and they have their own pooler

AAJR Hey all, can you confirm if you're still seeing impact? I'm seeing issues in so...

ivan•3/16/25, 3:53 AM

I can test tomorrow, my timescaledb is indeed in virginia

AJR•3/16/25, 4:13 AM

DMing you now to collect some information

AJR•3/16/25, 5:07 AM

This was escalated as an incident, and has now been remediated. We are monitoring status to ensure things remain functioning as expected.

https://www.cloudflarestatus.com/

Cloudflare Status

Welcome to Cloudflare's home for real-time and historical data on system performance.

onetwoDom•3/16/25, 6:41 AM

I am afraid we were still having issues. I created a new Hyperdrive to the same db and it works fine. CORRECTION: the new hyperdrive worked fine at first, but then Client Disconnected errors started building up again

onetwoDom•3/16/25, 6:41 AM

Btw loving the new Invocations tab in Workers Logs

AJR•3/16/25, 6:48 AM

Confirmed we saw impact start climbing again. Still working through it.

AJR•3/16/25, 8:09 AM

Impact has been resolved. We will monitor for a while yet.

AJR•3/16/25, 3:20 PM

Monitoring is showing impact picking back up, working to remediate this.

kchro3•3/16/25, 7:38 PM

when things have settled, can you share more about what has been causing these client disconnected issues over the past couple weeks? i was considering moving to a privately accessible database with a Hyperdrive connection, but i can't justify the risk now because at least with a publicly accessible db, i can connect to it normally.

AAJR Monitoring is showing impact picking back up, working to remediate this.

AJR•3/16/25, 11:02 PM

Things look mostly resolved, there's some backlogged traffic to work through, but if you're still seeing high error rates please let me know.

Kkchro3 when things have settled, can you share more about what has been causing these c...

AJR•3/16/25, 11:06 PM

There's a few different things contributing to this, and picking apart which are the worst offenders requires some observability improvements that should have gone out today. I'll try to set aside some time this week to write up our plans on this topic.

AAJR Things look mostly resolved, there's some backlogged traffic to work through, bu...

shay•3/16/25, 11:27 PM

i'm now getting a few s instead of timeouts

shay•3/16/25, 11:27 PM

not super high but i'm only sampling at 50%

shay•3/16/25, 11:28 PM

Definitely not every invocation though, in fact it seems pretty much fine from the user side

AJR•3/16/25, 11:28 PM

Well that's a good start, at least. Please send me your Hyperdrive ID so I can take a look

shay•3/16/25, 11:31 PM

9568cd870bee47f3801c862de747ca94

ivan•3/17/25, 12:02 AM

I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

Iivan I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

yevgen•3/17/25, 8:16 AM

The issue should be resolved by now. We identified the root cause and rolled out a hotfix.

kchro3•3/17/25, 11:54 AM

ramped traffic back up, so far ok

Original message was deleted

AJR•3/18/25, 1:38 PM

Hyperdrive connects to external databases (e.g Neon, RDS, on-prem Postgres, etc), so it would depend where and how you're hosting the database you're trying to connect to.

Hello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's been going great but r

Similar Threads

Similar Threads

Similar Threads