I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

Iivan I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

Y

yevgen•3/17/25, 8:16 AM

The issue should be resolved by now. We identified the root cause and rolled out a hotfix.

K

kchro3•3/17/25, 11:54 AM

ramped traffic back up, so far ok

Original message was deleted

A

AJR•3/18/25, 1:38 PM

Hyperdrive connects to external databases (e.g Neon, RDS, on-prem Postgres, etc), so it would depend where and how you're hosting the database you're trying to connect to.

A

AJR•3/18/25, 1:48 PM

That makes sense. Let me explain a bit.

D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.

Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.

A

AJR•3/18/25, 2:03 PM

It should be less than a couple seconds.

A

AJR•3/18/25, 2:03 PM

Getting all the way from WEUR to Bangkok might take slightly longer. If you observe longer than a minute, please let me know.

A

AJR•3/18/25, 2:03 PM

Even a minute would be out of line.

A

AJR•3/18/25, 2:16 PM

Cache hits for Hyperdrive will be 2-7ms latency. It won't be hard to tell once it gets to working.

A

AJR•3/18/25, 2:22 PM

Happy to. Good luck!

Original message was deleted

T

thomasgauvin•3/18/25, 2:26 PM

Let us know how it goes @PatrickJ! Also, depending on how cache heavy your workloads are you may find workers smart placement interesting. If you have more cold reads/less cache hits and a lot of subrequests per worker invocation, it may be best to have that worker placed near your data source

https://developers.cloudflare.com/workers/configuration/smart-placement/

Cloudflare Docs

Smart Placement · Cloudflare Workers docs

Speed up your Worker application by automatically placing your workloads in an optimal location that minimizes latency.

A

AJR•3/18/25, 2:28 PM

It'll take some traffic to kick on, but what it'll do when run with Hyperdrive is eventually learn to run the Worker itself right next to your DB. That'll make cache misses way faster, but slow down cache hits. Whether that's an improvement will depend on how often you get cache hits, like Thomas said.

A

AJR•3/18/25, 2:30 PM

That one I'm not sure on, sorry.

K

kunal•3/18/25, 7:02 PM

Our Cloudflare Worker (backed by Hyperdrive) had a big spike in Errors and Wall Time today starting at around 10:30am PT today. On Hyperdrive, I don't see any spikes in latency, but I did see a couple errors in each of our Hyperdrive instances at ~10:30am PT. Struggling a little with how to debug or fix this - most traffic is fine, but our P999 wall time jumped to 70k ms. All of our backing databases are completely normal, and I'm able to query Hyperdrive normally locally

Original message was deleted

E

Emilio•3/19/25, 10:08 PM

Yes in the context! Something like this middleware is recommended

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

// Middleware to inject the database client into the context
export const database = async (c: Context, next: Next) => {
  if (c.env.ENV === "test") {
    const database = new MockDatabaseClient();
    c.set("database", database);
    await next();
  } else {
    const sql = postgres(c.env.HYPERDRIVE.connectionString);
    const database = new KVDatabaseClient(sql);
    c.set("database", database); // set the database client to the context
    await next();
    // clean up the client ensuring we don't kill the worker before that is completed
    c.executionCtx.waitUntil(sql.end());
  }
};

Note that I'm wrapping the sql connection in another class for mocking purposes

EEmilio Yes in the context! Something like this middleware is recommended ```ts // Middl...

I

Isaac McFadyen•3/19/25, 11:10 PM

Is there a need to create+close the connection on every request here, given that Hyperdrive does pooling and so it wouldn't exhaust connection limits?

EEmilio Yes in the context! Something like this middleware is recommended ```ts // Middl...

D

DaniFoldi•3/19/25, 11:18 PM

I'd also add a maximum pool size to the postgres<->hyperdrive part, since >6 queries can deadlock the worker

IIsaac McFadyen Is there a need to create+close the connection on every request here, given that...

A

AJR•3/20/25, 10:56 AM

Blog coming soon. TLDR: that's a different "kind" of connection than the ones to the origin, takes ~2ms to open, and closing it will avoid holding your Worker open doing any processing, helping keep your CPU time down. I'd recommend doing it the way Emilio showed.

A

AJR•3/20/25, 10:57 AM

Send me the Hyperdrive ID, I'll take a look

A

AJR•3/20/25, 10:57 AM

Was it serving queries successfully at all?

A

AJR•3/20/25, 10:58 AM

One sec

A

AJR•3/20/25, 11:01 AM

Ok, moving to DMs so I can collect some information

A

AJR•3/20/25, 11:02 AM

I do see your connection errors

J

James•3/20/25, 6:45 PM

Hyperdrive got a nice shoutout in our case study with Neon today :MeowHeartCloudflare: https://neon.tech/blog/nodecraft-cloudflare-neon

Neon

How Nodecraft Built a Scalable Game Server Platform with Cloudflare...

To improve their scalability and performance, Nodecraft transitioned to a serverless architecture built on Cloudflare and Neon.

A

AJR•3/20/25, 9:05 PM

Nice! That got posted around a bit internally before I came to check here. Love to see it!

Original message was deleted

I

Isaac McFadyen•3/20/25, 10:38 PM

?crossposting

S

SuperHelpflare•3/20/25, 10:38 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

I

Isaac McFadyen•3/20/25, 10:39 PM

@System/Library/ServerSetup/ I've deleted your message as this has no relation to either KV or Hyperdrive.

S

Silvan•3/21/25, 10:08 AM

oh? we getting there

A

AJR•3/21/25, 12:32 PM

Yep. SoonTm

S

Silvan•3/21/25, 6:22 PM

Highly interested in seeing Planetscale with hyperdrive

SSilvan oh? we getting there 👀

I

Isaac McFadyen•3/21/25, 6:59 PM

You've discovered the secret of Cloudflare :lul:

S

Silvan•3/21/25, 7:00 PM

i have been stalking the changelogs for a while hahaha. not me asking for methods added 4 hours ago with no docs yet

SSilvan Highly interested in seeing Planetscale with hyperdrive

A

AJR•3/21/25, 7:00 PM

>.>

MySQL is going to be in beta for a while, but Planetscale are definitely on our radar.

S

Silvan•3/21/25, 7:03 PM

Yea makes sense. I mean Planetscale is the best cloud db solution IMO and probably 90% of people asking for mysql support

SSilvan i have been stalking the changelogs for a while hahaha. not me asking for method...

I

Isaac McFadyen•3/21/25, 7:25 PM

What you've got to stalk is the PRs for

workerd

workerd

:lul:

I

Isaac McFadyen•3/21/25, 7:25 PM

They're even more revealing than the changelogs, sometimes.

S

Silvan•3/21/25, 7:28 PM

i do i do even the agents repo but that one dosent have changelogs so commits it is :^)

AAJR There's a few different things contributing to this, and picking apart which are...

K

kchro3•3/21/25, 7:29 PM

Any chance you have an update on this? Still curious to hear about what happened

Kkchro3 Any chance you have an update on this? Still curious to hear about what happened

A

AJR•3/21/25, 8:01 PM

Haven't forgotten, but we're actively investigating some of the moving parts. Here's what we have so far.

Recent incidents causing disconnects: a bug in some users' scripts have identified a noisy neighbor risk in our service. We're working to remedy that. A couple other smaller spikes have walked through the same ground
Observability gaps: the increase in disconnects from the above point has identified some gaps in our observability and error reporting. We've released some improvements to this, and are planning to work on more.

A

AJR•3/21/25, 8:07 PM

I do want to stress that the users in question were not malicious, this wasn't a DOS attack because I said something in discord. Honest mistake that highlighted a gap in our hardening we hadn't spotted, that we now need to address.

A

AJR•3/21/25, 8:14 PM

As we learn more, and are able to roll out improvements, I'll post another followup here.

J

Jökull Sólberg•3/25/25, 7:23 PM

Please make cache off by default. Took me a while to debug some stuff. Also maybe a security issue?

JJökull Sólberg Please make cache off by default. Took me a while to debug some stuff. Also mayb...

D

DaniFoldi•3/25/25, 8:24 PM

@thomasgauvin I second this, I remember also running in to this when I set up my first hyperdrive
or prompt the users if they have not specified --cache=true or =false, something like that

AAJR >.> MySQL is going to be in beta for a while, but Planetscale are definitely on...

A

antim8•3/25/25, 9:49 PM

Are there any estimates on when mysql is going to be in beta?

DDaniFoldi @thomasgauvin I second this, I remember also running in to this when I set up my...

A

AJR•3/25/25, 10:15 PM

While I would certainly entertain the idea that would have been a better default, I can't imagine we'd flip that behavior at this stage without a really strong reason to do so.

Aantim8 Are there any estimates on when mysql is going to be in beta?

A

AJR•3/25/25, 10:16 PM

We don't generally publish dates like that publicly, it doesn't usually help anyone and can cause issues if something delays us. I will say that it's actually soon, and not like months away or anything.

DDaniFoldi @thomasgauvin I second this, I remember also running in to this when I set up my...

A

AJR•3/25/25, 10:18 PM

A prompt is an interesting idea. I'll chat about that with the team, see how folks feel.

AAJR A prompt is an interesting idea. I'll chat about that with the team, see how fo...

J

Jökull Sólberg•3/26/25, 9:19 AM

If the default will remain unchanged, I would propose the following changes for an improved onboarding experience with fewer gotchas.

Update "Getting started" wording - currently it says "Hyperdrive [...] can cache the most common read queries, improving performance[...].". To me this strongly suggests that it defaults to off and is opt in.
Add a note to "Getting started" where the user is instructed to run
```
wrangler hyperdrive create
```
wrangler hyperdrive create
```
wrangler hyperdrive create
```
wrangler hyperdrive create
-- something like "To disable and opt out of caching add
```
--caching-disabled
```
--caching-disabled
```
--caching-disabled
```
--caching-disabled
. You can also specify
```
--max-age
```
--max-age
```
--max-age
```
--max-age
in seconds before query results are evicted (default 60 seconds)". Maybe the word "caching" would be linked to the cache docs.

I did notice that the

--help

--help

--help

--help has great documentation btw. I should have read through that before issuing the command

JJökull Sólberg If the default will remain unchanged, I would propose the following changes for ...

A

AJR•3/26/25, 11:41 AM

I think we can definitely add that language to the docs. Appreciate the suggestions!

AAJR A prompt is an interesting idea. I'll chat about that with the team, see how fo...

T

thomasgauvin•3/26/25, 1:51 PM

I think we could do this, yes, in the dashboard and/or in Wrangler

I'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT

Similar Threads