I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would e

I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would expect this to result in a error and not the working hanging once the TCP streams limit is hit? I have since also lowered that to just 1 to be sure.

AAlex I had the default settings mysql2/drizzle set which technically cna be 10 I thin...

Isaac McFadyen•7/7/25, 9:48 PM

Theoretically the connections are queued. The issue is that new connections are queued, but old ones can potentially be held open waiting on new ones, so a deadlock occurs.

Isaac McFadyen•7/7/25, 9:48 PM

And then the runtime detects that deadlock and throws, assuming that's the issue here (it very well might not be)

AlexOP•7/7/25, 9:50 PM

I see, got it. So that could indeed in theory be a cause then would depend on how long the isolate lives for it to create that many connections though. My connection pool lives inside the

fetch

fetch

handler so I would expect a new "pool" every request and the old one discarded but who knows what it actually does under the hood

AlexOP•7/8/25, 8:13 PM

I am seeing something else strange. I am using Drizzle with mysql2 and sometimes queries return no results and also no error but are also unable to find a row. This result seems to be cached by Hyperdrive because the effects can be seen for a few seconds. I have never seen MySQL return a wrong result. I know these rows exist, and they are not modified or deleted/re-inserted or whatever. I do see a error in Hyperdrive around that time, but I don't think there is a way to correlate those errors to a specific request right now.

I run a select with limit(1) so I expect either a empty list or 1 result. So I check this with result.length === 0 and return a 404 response. However sometimes this 404 response is thus incorrect. I can see from my own logging the code thinks there is no result even though there should be one (or an error).

This might very well be a Drizzle issue but I was wondering if this might sound like something anyone has seen before or can point me in the right direction.

AJR•7/8/25, 8:17 PM

Hmmm. We do very, very little messing with the packet structure in our mysql handler, but we deserialize it, so it's not technically impossible for that to be related to Hyperdrive.

To make sure I understand:

you send a query that's some variant of
```
SELECT x, y, z FROM myTable LIMIT 1
```
```
SELECT x, y, z FROM myTable LIMIT 1
```
, and get a malformed response where you get 0 rows returned when you'd have expected 1 row?
this 0-length response is then cached, so you see (as you'd expect) the 0-row result persisted as cache hits for your TTL
on subsequent queries, when cache is refreshed, you get the 1 row you'd have expected

That correctly summarize what you're seeing?

AlexOP•7/8/25, 8:19 PM

Yes, this is exactly it. This is also intermittent. I have seen this happen at least 10 times between 18:30 and 19:30 (that I know I can filter in the logs). Not sure how many of those 10 were cached responses ofcourse but this happens from time to time.

AlexOP•7/8/25, 8:22 PM

I should also mention this comes in spurts. It happens a few times and then not for many hours. I was thinking this was just the uptime monitoring software acting up, but I've seen it happen a few times now and even lowered the cache TTL because of it to lessen the length of the impact.

AlexOP•7/8/25, 8:23 PM

If it helps I can share some request IDs, unsure that is of any use but I can make a list.

AJR•7/8/25, 8:23 PM

Yes, a few request IDs would be useful so we can take a look

AJR•7/8/25, 8:24 PM

Thank you

AJR•7/8/25, 8:25 PM

I'll add a ticket to our MySQL group to take a look at this also. Appreciate you mentioning it.

AlexOP•7/8/25, 8:26 PM

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

AlexOP•7/8/25, 8:27 PM

The exact query (in case it matters):

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

First parameter is

example.com

example.com

(string) and second

(number)

AJR•7/8/25, 8:28 PM

Got it. I would expect that that shouldn't matter, but until we have a root cause more info is always helpful.

AlexOP•7/10/25, 12:50 PM

FYI; this got so bad just now that I've disabled my worker for now. This sporadic nature is terrible for a links shortener as you can imagine

Giving people a 20% odds they hit a 404 instead of their intended redirect is not a good USP I have been told.

AJR•7/10/25, 12:51 PM

Hmmm. We haven't released any changes at all to mysql handler in about a month.

AJR•7/10/25, 12:51 PM

Okay, thanks for the heads up. That's unfortunate

AlexOP•7/10/25, 12:53 PM

Maybe it was always this bad and customers just started complaining but I don't know. I need to implement some checks and race my origin response vs the worker response and see how often they differ. But something is definitly broken for me.

AJR•7/10/25, 12:55 PM

Yeah I dug through a bit yesterday and haven't been able to repro 0-length results when data is present

AJR•7/10/25, 12:56 PM

Going to dig some more with your stack included, drizzle is popular enough that I expect it would behave well

AlexOP•7/10/25, 1:01 PM

I will try and see if I can figure out a way to get some useful debug output.

AlexOP•7/10/25, 1:02 PM

But the

typeof

typeof

for a succesful query was both

object

object

although I know realize this was a pretty pointless thing to check for... adding some more console logs of the actual contents of the result.

AlexOP•7/10/25, 1:27 PM

Also slightly unrealted but not sure where to report this, the inputs on the logs page of a worker are pretty bright in dark mode (and the text is unreadable, there is something in there): https://screen.bouma.link/5xfP7KwLsj0k7wxxKXgw
Same goes for the global observability pages under the workers menu item

CleanShot Cloud

Screenshot 2025-07-10 at 15.26.22

Screenshot

AAlex But the `typeof` for a succesful query was both `object` although I know realize...

AlexOP•7/10/25, 6:21 PM

So as it seems it is indeed the result is incorrect without an error. I either get a array with 1 response or an empty array. It looks like it happens anywhere from 1-100 times an hour. This is from a total of 10k requests. So it's like ~1% of requests. That is pretty high actually.

AAlex So as it seems it is indeed the result is incorrect without an error. I either g...

AJR•7/10/25, 6:23 PM

Thank you. I'm continuing to investigate, will keep you posted if I find something on our end.

kenpachi•7/13/25, 3:34 PM

Does hyperdrive not work with tunnels? I have been looking aorund for a while now i have seen a lot of users facing issues like Failed to connect to the provided database: Connecting to database via Cloudflare Tunnel failed: 403 Forbidden.
I am using coolify and I have followed the docs in setting the tunnels and they are up and runing, but cant connect to my db over tcp

Kkenpachi Does hyperdrive not work with tunnels? I have been looking aorund for a while n...

AJR•7/13/25, 4:50 PM

Hyperdrive does work with tunnels, yes, but also requires an access policy on the tunnel for security.

403 means there's an issue with some component of setting up the Access Policy on the tunnel.

Coolify's automatic SSL setup uses a custom curve for the encryption that Hyperdrive doesn't currently support, so you may have to deal with TLS on your origin DB manually to get it working.

AAJR Hyperdrive does work with tunnels, yes, but also requires an access policy on th...

kenpachi•7/13/25, 5:28 PM

I already set the cloudflare tunnel to access all my resources, and my dns and ssl is managed by cloudflare, the issue has been a blocker

AJR•7/13/25, 5:31 PM

Sorry which issue is that?

AJR•7/13/25, 5:34 PM

A 403 sounds like you'll need to handle the access policy misconfiguration.

For SSL on your origin DB, Cloudflare does not (cannot) manage that. See this issue for more details: https://github.com/cloudflare/workers-sdk/issues/8671

GitHub

🐛 BUG: Cannot connect to Hyperdrive from Coolify PostgreSQL Data...

Which Cloudflare product(s) does this pertain to? Other What versions are you using? 4.4.0 [Wrangler], 16.8 [PostgreSQL] What operating system and version are you using? Windows 10 21H2 (Wrangler) ...

AAJR Sorry which issue is that?

kenpachi•7/13/25, 6:12 PM

I keep getting Failed to connect to the provided database: Connecting to database via Cloudflare Tunnel failed: 503 Service Unavailable, only the db

TeamDave•7/15/25, 2:38 PM

Hello! I'm building out an analytics tool that makes a lot of db calls (10+ calls to get the things like total users, unique sessions, events, etc).

I'm using cloudflare pages, so locally, it's using a sqlite db and the requests can take up to 2 seconds. When I go to staging it uses the D1 db and my requests can take up to 3 minutes.

Any ideas why it would be so different? I'm thinking maybe the db connections are what's taking so long, I am making a lot of connections but don't want to spend the time trying to combine the sql if it's not going to get me down to that ~2 second mark.

TTeamDave Hello! I'm building out an analytics tool that makes a lot of db calls (10+ call...

AJR•7/15/25, 2:40 PM

I think you might get better luck in #d1-database . I'm afraid I'm not as up to speed on the internals there.

I will say that it definitely shouldn't take 3 minutes to do 10 queries, regardless of anything else, so I'm optimistic there's a solution there.

AAJR I think you might get better luck in #d1-database . I'm afraid I'm not as up to ...

TeamDave•7/15/25, 2:40 PM

Man... I thought that's where I was, thanks!

ALECKS•7/17/25, 2:25 PM

Why doesn't the hyperdrive binding contain the connection details without having to call

.get()

.get()

? This is adds significant friction to the DX.

We're using a lot of RCP/Service workers, and it would be so natural to configure/initialize database stuff in a constructor and attach it to

this.

this.

Then every public method that could be invoked via RPC could simply use

this.db...

this.db...

. But because

.get()

.get()

is async, we need to do this dance in every public method.

AALECKS Why doesn't the hyperdrive binding contain the connection details without having...

Isaac McFadyen•7/17/25, 2:31 PM

where/how are you using

get()

get()

? Hyperdrive connection details should be on the binding itself, according to docs?

IIsaac McFadyen 🤔 where/how are you using `get()`? Hyperdrive connection details should be on t...

ALECKS•7/17/25, 2:34 PM

I'm sorry, my statement was incorrect. There's nothing wrong with the hyperdrive binding. The issue is with the secrets_store binding and I got them mixed up. Will repost in the right channel

karimulm•7/17/25, 7:10 PM

Is hyperdrive accessible from cloudflare container?

AJR•7/17/25, 9:04 PM

Not at this time, but it's something we intend to add as it's an obviously strong use case for both products.

AAJR Not at this time, but it's something we intend to add as it's an obviously stron...

ALECKS•7/17/25, 9:43 PM

Would it be okay to get the connectionString in a worker and pass it to a container or is that not advised?

AJR•7/17/25, 9:45 PM

I'm afraid it won't work. The connection string provided by the Hyperdrive binding is generated at runtime and is scoped to just that request. It won't work anywhere else or at any other time.

AJR•7/17/25, 9:45 PM

Part of the security model.

AAJR I'm afraid it won't work. The connection string provided by the Hyperdrive bindi...

Isaac McFadyen•7/18/25, 12:35 AM

Interesting, if a connection is kept open (i.e. a global connection against recommendations of closing it at the end of a request) will it continue to work until it's closed then?

IIsaac McFadyen Interesting, if a connection is kept open (i.e. a global connection against reco...

AJR•7/18/25, 12:54 AM

I'm not an expert in how the Worker Runtime manages state in edge cases like that, honestly. What I know for sure is that when the connection goes the connectionString goes with it

andrzej•7/18/25, 10:45 AM

Hi! I'm working on some Worker code which needs to fetch about a 1000 of small rows from a MySQL database (hosted by PlanetScale). When I connect using Hyperdrive, the CPU usage per request is way higher than when connecting directly. For example, 5ms with a direct connection, compared to 40ms with Hyperdrive.
The root cause - to my understanding - is the fact that when receiving data from Hyperdrive, each resulting row (mysql protocol packet) is returned in what appears to be a separate TCP packet. In my tests, each call to ReadableStreamBYOBReader.read returns only 22 bytes when receiving query results (this matches the size of the mysql packet), despite having lots of free space in the buffer.
I find this surprising, because conventional TCP mechanisms such as Nagle's algorithm should merge these small packets. If my understanding is correct, then this behaviour also introduces inefficiencies at other OS/network layers.
For comparison, when I connect directly to planetscale, the data is returned in much larger chunks (multiple kilobytes), allowing for more efficient processing.
A dirty workaround is to simply add a delay before receiving the query result. This allows one read() call to process a larger chunk of data.

AJR•7/18/25, 11:07 AM

That's so interesting.

The underlying cause here is that under the hood Hyperdrive isn't TCP except for the very last hop to/from the origin database. The overwhelming majority of the transit uses different transport.

I'll make a ticket to investigate our batching to see if we can't improve it to at least parity with a direct connection. We focused far more on latency and throughput for the beta, and our Postgres handler has a bunch of other buffering built in by nature of the protocol. However we definitely don't want to be running up CPU time on MySQL either.

Thanks for bringing this to our attention.

AJR•7/18/25, 11:07 AM

Fyi @thomasgauvin ^

Aandrzej Hi! I'm working on some Worker code which needs to fetch about a 1000 of small r...

AJR•7/18/25, 3:35 PM

Which drivers are you using for Hyperdrive and the direct connection to Planetscale?

andrzej•7/18/25, 4:12 PM

I've started with mysql2, but then I've moved to a handcrafted mysql protocol implementation, hoping to avoid any unnecessary CPU overheads.

Aandrzej I've started with mysql2, but then I've moved to a handcrafted mysql protocol im...

AJR•7/18/25, 4:14 PM

Sounds good. Wanted to make sure that the difference was visible even with the same driver, to rule out having to explore driver-level differences also. Thank you!

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would e

Similar Threads