Monitoring is showing impact picking back up, working to remediate this.

kchro3•3/16/25, 7:38 PM

when things have settled, can you share more about what has been causing these client disconnected issues over the past couple weeks? i was considering moving to a privately accessible database with a Hyperdrive connection, but i can't justify the risk now because at least with a publicly accessible db, i can connect to it normally.

AAJR Monitoring is showing impact picking back up, working to remediate this.

AJROP•3/16/25, 11:02 PM

Things look mostly resolved, there's some backlogged traffic to work through, but if you're still seeing high error rates please let me know.

Kkchro3 when things have settled, can you share more about what has been causing these c...

AJROP•3/16/25, 11:06 PM

There's a few different things contributing to this, and picking apart which are the worst offenders requires some observability improvements that should have gone out today. I'll try to set aside some time this week to write up our plans on this topic.

AAJR Things look mostly resolved, there's some backlogged traffic to work through, bu...

shay•3/16/25, 11:27 PM

i'm now getting a few CONNECTION_CLOSEDCONNECTION_CLOSEDs instead of timeouts

shay•3/16/25, 11:27 PM

not super high but i'm only sampling at 50%

shay•3/16/25, 11:28 PM

Definitely not every invocation though, in fact it seems pretty much fine from the user side

AJROP•3/16/25, 11:28 PM

Well that's a good start, at least. Please send me your Hyperdrive ID so I can take a look

shay•3/16/25, 11:31 PM

9568cd870bee47f3801c862de747ca94

ivan•3/17/25, 12:02 AM

I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

Iivan I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

yevgen•3/17/25, 8:16 AM

The issue should be resolved by now. We identified the root cause and rolled out a hotfix.

kchro3•3/17/25, 11:54 AM

ramped traffic back up, so far ok

Original message was deleted

AJROP•3/18/25, 1:38 PM

Hyperdrive connects to external databases (e.g Neon, RDS, on-prem Postgres, etc), so it would depend where and how you're hosting the database you're trying to connect to.

AJROP•3/18/25, 1:48 PM

That makes sense. Let me explain a bit.

D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.

Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.

AJROP•3/18/25, 2:03 PM

It should be less than a couple seconds.

AJROP•3/18/25, 2:03 PM

Getting all the way from WEUR to Bangkok might take slightly longer. If you observe longer than a minute, please let me know.

AJROP•3/18/25, 2:03 PM

Even a minute would be out of line.

AJROP•3/18/25, 2:16 PM

Cache hits for Hyperdrive will be 2-7ms latency. It won't be hard to tell once it gets to working.

AJROP•3/18/25, 2:22 PM

Happy to. Good luck!

Original message was deleted

thomasgauvin•3/18/25, 2:26 PM

Let us know how it goes @PatrickJ! Also, depending on how cache heavy your workloads are you may find workers smart placement interesting. If you have more cold reads/less cache hits and a lot of subrequests per worker invocation, it may be best to have that worker placed near your data source

https://developers.cloudflare.com/workers/configuration/smart-placement/

Cloudflare Docs

Smart Placement · Cloudflare Workers docs

Speed up your Worker application by automatically placing your workloads in an optimal location that minimizes latency.

AJROP•3/18/25, 2:28 PM

It'll take some traffic to kick on, but what it'll do when run with Hyperdrive is eventually learn to run the Worker itself right next to your DB. That'll make cache misses way faster, but slow down cache hits. Whether that's an improvement will depend on how often you get cache hits, like Thomas said.

AJROP•3/18/25, 2:30 PM

That one I'm not sure on, sorry.

kunal•3/18/25, 7:02 PM

Our Cloudflare Worker (backed by Hyperdrive) had a big spike in Errors and Wall Time today starting at around 10:30am PT today. On Hyperdrive, I don't see any spikes in latency, but I did see a couple errors in each of our Hyperdrive instances at ~10:30am PT. Struggling a little with how to debug or fix this - most traffic is fine, but our P999 wall time jumped to 70k ms. All of our backing databases are completely normal, and I'm able to query Hyperdrive normally locally

Original message was deleted

Emilio•3/19/25, 10:08 PM

Yes in the context! Something like this middleware is recommended

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

Note that I'm wrapping the sql connection in another class for mocking purposes

EEmilio Yes in the context! Something like this middleware is recommended ```ts // Middl...

Isaac McFadyen•3/19/25, 11:10 PM

Is there a need to create+close the connection on every request here, given that Hyperdrive does pooling and so it wouldn't exhaust connection limits?

EEmilio Yes in the context! Something like this middleware is recommended ```ts // Middl...

DaniFoldi•3/19/25, 11:18 PM

I'd also add a maximum pool size to the postgres<->hyperdrive part, since >6 queries can deadlock the worker

IIsaac McFadyen Is there a need to create+close the connection on every request here, given that...

AJROP•3/20/25, 10:56 AM

Blog coming soon. TLDR: that's a different "kind" of connection than the ones to the origin, takes ~2ms to open, and closing it will avoid holding your Worker open doing any processing, helping keep your CPU time down. I'd recommend doing it the way Emilio showed.

AJROP•3/20/25, 10:57 AM

Send me the Hyperdrive ID, I'll take a look

AJROP•3/20/25, 10:57 AM

Was it serving queries successfully at all?

AJROP•3/20/25, 10:58 AM

One sec

AJROP•3/20/25, 11:01 AM

Ok, moving to DMs so I can collect some information

AJROP•3/20/25, 11:02 AM

I do see your connection errors

James•3/20/25, 6:45 PM

Hyperdrive got a nice shoutout in our case study with Neon today :MeowHeartCloudflare: https://neon.tech/blog/nodecraft-cloudflare-neon

Neon

How Nodecraft Built a Scalable Game Server Platform with Cloudflare...

To improve their scalability and performance, Nodecraft transitioned to a serverless architecture built on Cloudflare and Neon.

AJROP•3/20/25, 9:05 PM

Nice! That got posted around a bit internally before I came to check here. Love to see it!

Original message was deleted

Isaac McFadyen•3/20/25, 10:38 PM

?crossposting

SuperHelpflare•3/20/25, 10:38 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Isaac McFadyen•3/20/25, 10:39 PM

@System/Library/ServerSetup/ I've deleted your message as this has no relation to either KV or Hyperdrive.

Silvan•3/21/25, 10:08 AM

oh? we getting there

AJROP•3/21/25, 12:32 PM

Yep. SoonTm

Silvan•3/21/25, 6:22 PM

Highly interested in seeing Planetscale with hyperdrive

SSilvan oh? we getting there 👀

Isaac McFadyen•3/21/25, 6:59 PM

You've discovered the secret of Cloudflare :lul:

Silvan•3/21/25, 7:00 PM

i have been stalking the changelogs for a while hahaha. not me asking for methods added 4 hours ago with no docs yet

SSilvan Highly interested in seeing Planetscale with hyperdrive

AJROP•3/21/25, 7:00 PM

>.>

MySQL is going to be in beta for a while, but Planetscale are definitely on our radar.

Silvan•3/21/25, 7:03 PM

Yea makes sense. I mean Planetscale is the best cloud db solution IMO and probably 90% of people asking for mysql support

SSilvan i have been stalking the changelogs for a while hahaha. not me asking for method...

Isaac McFadyen•3/21/25, 7:25 PM

What you've got to stalk is the PRs for workerdworkerd :lul:

Isaac McFadyen•3/21/25, 7:25 PM

They're even more revealing than the changelogs, sometimes.

Silvan•3/21/25, 7:28 PM

i do i do even the agents repo but that one dosent have changelogs so commits it is :^)

AAJR There's a few different things contributing to this, and picking apart which are...

kchro3•3/21/25, 7:29 PM

Any chance you have an update on this? Still curious to hear about what happened

Kkchro3 Any chance you have an update on this? Still curious to hear about what happened

AJROP•3/21/25, 8:01 PM

Haven't forgotten, but we're actively investigating some of the moving parts. Here's what we have so far.

Recent incidents causing disconnects: a bug in some users' scripts have identified a noisy neighbor risk in our service. We're working to remedy that. A couple other smaller spikes have walked through the same ground
Observability gaps: the increase in disconnects from the above point has identified some gaps in our observability and error reporting. We've released some improvements to this, and are planning to work on more.

AJROP•3/21/25, 8:07 PM

I do want to stress that the users in question were not malicious, this wasn't a DOS attack because I said something in discord. Honest mistake that highlighted a gap in our hardening we hadn't spotted, that we now need to address.

AJROP•3/21/25, 8:14 PM

As we learn more, and are able to roll out improvements, I'll post another followup here.

// Middleware to inject the database client into the context export const database = async (c: Context, next: Next) => { if (c.env.ENV === "test") { const database = new MockDatabaseClient(); c.set("database", database); await next(); } else { const sql = postgres(c.env.HYPERDRIVE.connectionString); const database = new KVDatabaseClient(sql); c.set("database", database); // set the database client to the context await next(); // clean up the client ensuring we don't kill the worker before that is completed c.executionCtx.waitUntil(sql.end()); } };

Monitoring is showing impact picking back up, working to remediate this.

Similar Threads

Similar Threads

Similar Threads