So I have been having some sporadic issues. And I apologize for my vague "accusations" but I believe

So I have been having some sporadic issues. And I apologize for my vague "accusations" but I believe this might be caused by Hyperdrive or drizzle or mysql2 or some combination thereof. I have been seeing spikes of requests that receive the following error:

The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/

or this one:

Promise will never complete.

The requests timeout and fail obviously but the worker is stupid simple (it fails at 3ms CPU time and 60s wall time so it's obviously waiting for something external) and I have never seen this error on "normal" workers (those not interacting with Hyperdrive) before.

According to my Hyperdrive dashboard there were 2 errors recorded, but they don't seem to be around the time the problems happen. Also just 2 even though thousands requests failed in a short period so something somewhere is going wrong and I'm not sure where to look.

I have just tried to add timeouts everywhere I can think of but if anyone has any other idea I'm all ears

AAlex So I have been having some sporadic issues. And I apologize for my vague "accusa...

AJR•7/7/25, 8:36 PM

When did it start? Is it still ongoing (even if sporadically)?

AlexOP•7/7/25, 8:42 PM

According to a client for about a week maybe 2, but honestly I am unsure. And yes this still seems to be ongoing.

AAlex According to a client for about a week maybe 2, but honestly I am unsure. And ye...

AJR•7/7/25, 8:43 PM

Okay, I'm passing this along to my team right now, and we'll keep you posted on what we find. I'll have to reach out to the runtime team to see if we can correlate whether this is happening on other Hyperdrive-enabled workers, and what might be going on here.

If you have any other information/logs/etc that you think might be relevant please feel free to send it my way either here or via DM.

AlexOP•7/7/25, 8:45 PM

I'll try and send you anything I can find that might be helpful, appreciate you looking into it.

AlexOP•7/7/25, 8:52 PM

If we assume queries from my server take single digit ms to execute. Is it fair to assume that worldwide queries should complete within 250ms using Hyperdrive in most cases? Always exceptions ofcourse but I am trying to figure out if my timeout is too aggresive. According to the Hyperdrive dashboard latency is 31 ms past 24 hours, but unsure if that is worldwide.

AAlex If we assume queries from my server take single digit ms to execute. Is it fair ...

AJR•7/7/25, 9:03 PM

It's so hard to say, networking and other factors make any one number I could give you either too large to be useful or likely to be wrong sometimes, I'm afraid

AAlex I'll try and send you anything I can find that might be helpful, appreciate you ...

AJR•7/7/25, 9:04 PM

Absolutely happy to.

And please don't worry about "accusations". We are incredibly grateful to have users who are willing to go to the trouble of flagging potential problems for us. You're not going to hurt our feelings if it turns out to be something unrelated.

Isaac McFadyen•7/7/25, 9:07 PM

Out of curiosity do you specify max_connections (or whatever MySQL calls it) when creating your client?

Isaac McFadyen•7/7/25, 9:07 PM

It reminds of when I've seen this happen before with Postgres when it tries to create more than 6 connections (which fails because Workers limits TCP streams to 6 maximum)...

Isaac McFadyen•7/7/25, 9:08 PM

Which resulted in this docs issue https://github.com/cloudflare/cloudflare-docs/pull/19616 and this escalation thread (for AJR, the channel is private) https://discord.com/channels/595317990191398933/1334906924432490586/1334906924432490586

AlexOP•7/7/25, 9:20 PM

I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would expect this to result in a error and not the working hanging once the TCP streams limit is hit? I have since also lowered that to just 1 to be sure.

AAlex I had the default settings mysql2/drizzle set which technically cna be 10 I thin...

Isaac McFadyen•7/7/25, 9:48 PM

Theoretically the connections are queued. The issue is that new connections are queued, but old ones can potentially be held open waiting on new ones, so a deadlock occurs.

Isaac McFadyen•7/7/25, 9:48 PM

And then the runtime detects that deadlock and throws, assuming that's the issue here (it very well might not be)

AlexOP•7/7/25, 9:50 PM

I see, got it. So that could indeed in theory be a cause then would depend on how long the isolate lives for it to create that many connections though. My connection pool lives inside the

fetch

fetch

handler so I would expect a new "pool" every request and the old one discarded but who knows what it actually does under the hood

AlexOP•7/8/25, 8:13 PM

I am seeing something else strange. I am using Drizzle with mysql2 and sometimes queries return no results and also no error but are also unable to find a row. This result seems to be cached by Hyperdrive because the effects can be seen for a few seconds. I have never seen MySQL return a wrong result. I know these rows exist, and they are not modified or deleted/re-inserted or whatever. I do see a error in Hyperdrive around that time, but I don't think there is a way to correlate those errors to a specific request right now.

I run a select with limit(1) so I expect either a empty list or 1 result. So I check this with result.length === 0 and return a 404 response. However sometimes this 404 response is thus incorrect. I can see from my own logging the code thinks there is no result even though there should be one (or an error).

This might very well be a Drizzle issue but I was wondering if this might sound like something anyone has seen before or can point me in the right direction.

AJR•7/8/25, 8:17 PM

Hmmm. We do very, very little messing with the packet structure in our mysql handler, but we deserialize it, so it's not technically impossible for that to be related to Hyperdrive.

To make sure I understand:

you send a query that's some variant of SELECT x, y, z FROM myTable LIMIT 1SELECT x, y, z FROM myTable LIMIT 1, and get a malformed response where you get 0 rows returned when you'd have expected 1 row?
this 0-length response is then cached, so you see (as you'd expect) the 0-row result persisted as cache hits for your TTL
on subsequent queries, when cache is refreshed, you get the 1 row you'd have expected

That correctly summarize what you're seeing?

AlexOP•7/8/25, 8:19 PM

Yes, this is exactly it. This is also intermittent. I have seen this happen at least 10 times between 18:30 and 19:30 (that I know I can filter in the logs). Not sure how many of those 10 were cached responses ofcourse but this happens from time to time.

AlexOP•7/8/25, 8:22 PM

I should also mention this comes in spurts. It happens a few times and then not for many hours. I was thinking this was just the uptime monitoring software acting up, but I've seen it happen a few times now and even lowered the cache TTL because of it to lessen the length of the impact.

AlexOP•7/8/25, 8:23 PM

If it helps I can share some request IDs, unsure that is of any use but I can make a list.

AJR•7/8/25, 8:23 PM

Yes, a few request IDs would be useful so we can take a look

AJR•7/8/25, 8:24 PM

Thank you

AJR•7/8/25, 8:25 PM

I'll add a ticket to our MySQL group to take a look at this also. Appreciate you mentioning it.

AlexOP•7/8/25, 8:26 PM

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

AlexOP•7/8/25, 8:27 PM

The exact query (in case it matters):

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

First parameter is

example.com

example.com

(string) and second

(number)

AJR•7/8/25, 8:28 PM

Got it. I would expect that that shouldn't matter, but until we have a root cause more info is always helpful.

AlexOP•7/10/25, 12:50 PM

FYI; this got so bad just now that I've disabled my worker for now. This sporadic nature is terrible for a links shortener as you can imagine

Giving people a 20% odds they hit a 404 instead of their intended redirect is not a good USP I have been told.

AJR•7/10/25, 12:51 PM

Hmmm. We haven't released any changes at all to mysql handler in about a month.

AJR•7/10/25, 12:51 PM

Okay, thanks for the heads up. That's unfortunate

AlexOP•7/10/25, 12:53 PM

Maybe it was always this bad and customers just started complaining but I don't know. I need to implement some checks and race my origin response vs the worker response and see how often they differ. But something is definitly broken for me.

AJR•7/10/25, 12:55 PM

Yeah I dug through a bit yesterday and haven't been able to repro 0-length results when data is present

AJR•7/10/25, 12:56 PM

Going to dig some more with your stack included, drizzle is popular enough that I expect it would behave well

AlexOP•7/10/25, 1:01 PM

I will try and see if I can figure out a way to get some useful debug output.

AlexOP•7/10/25, 1:02 PM

But the typeoftypeof for a succesful query was both

object

object

although I know realize this was a pretty pointless thing to check for... adding some more console logs of the actual contents of the result.

AlexOP•7/10/25, 1:27 PM

Also slightly unrealted but not sure where to report this, the inputs on the logs page of a worker are pretty bright in dark mode (and the text is unreadable, there is something in there): https://screen.bouma.link/5xfP7KwLsj0k7wxxKXgw
Same goes for the global observability pages under the workers menu item

CleanShot Cloud

Screenshot 2025-07-10 at 15.26.22

Screenshot

AAlex But the `typeof` for a succesful query was both `object` although I know realize...

AlexOP•7/10/25, 6:21 PM

So as it seems it is indeed the result is incorrect without an error. I either get a array with 1 response or an empty array. It looks like it happens anywhere from 1-100 times an hour. This is from a total of 10k requests. So it's like ~1% of requests. That is pretty high actually.

AAlex So as it seems it is indeed the result is incorrect without an error. I either g...

AJR•7/10/25, 6:23 PM

Thank you. I'm continuing to investigate, will keep you posted if I find something on our end.

kenpachi•7/13/25, 3:34 PM

Does hyperdrive not work with tunnels? I have been looking aorund for a while now i have seen a lot of users facing issues like Failed to connect to the provided database: Connecting to database via Cloudflare Tunnel failed: 403 Forbidden.
I am using coolify and I have followed the docs in setting the tunnels and they are up and runing, but cant connect to my db over tcp

Kkenpachi Does hyperdrive not work with tunnels? I have been looking aorund for a while n...

AJR•7/13/25, 4:50 PM

Hyperdrive does work with tunnels, yes, but also requires an access policy on the tunnel for security.

403 means there's an issue with some component of setting up the Access Policy on the tunnel.

Coolify's automatic SSL setup uses a custom curve for the encryption that Hyperdrive doesn't currently support, so you may have to deal with TLS on your origin DB manually to get it working.

AAJR Hyperdrive does work with tunnels, yes, but also requires an access policy on th...

kenpachi•7/13/25, 5:28 PM

I already set the cloudflare tunnel to access all my resources, and my dns and ssl is managed by cloudflare, the issue has been a blocker

AJR•7/13/25, 5:31 PM

Sorry which issue is that?

AJR•7/13/25, 5:34 PM

A 403 sounds like you'll need to handle the access policy misconfiguration.

For SSL on your origin DB, Cloudflare does not (cannot) manage that. See this issue for more details: https://github.com/cloudflare/workers-sdk/issues/8671

GitHub

🐛 BUG: Cannot connect to Hyperdrive from Coolify PostgreSQL Data...

Which Cloudflare product(s) does this pertain to? Other What versions are you using? 4.4.0 [Wrangler], 16.8 [PostgreSQL] What operating system and version are you using? Windows 10 21H2 (Wrangler) ...

AAJR Sorry which issue is that?

kenpachi•7/13/25, 6:12 PM

I keep getting Failed to connect to the provided database: Connecting to database via Cloudflare Tunnel failed: 503 Service Unavailable, only the db

TeamDave•7/15/25, 2:38 PM

Hello! I'm building out an analytics tool that makes a lot of db calls (10+ calls to get the things like total users, unique sessions, events, etc).

I'm using cloudflare pages, so locally, it's using a sqlite db and the requests can take up to 2 seconds. When I go to staging it uses the D1 db and my requests can take up to 3 minutes.

Any ideas why it would be so different? I'm thinking maybe the db connections are what's taking so long, I am making a lot of connections but don't want to spend the time trying to combine the sql if it's not going to get me down to that ~2 second mark.

TTeamDave Hello! I'm building out an analytics tool that makes a lot of db calls (10+ call...

AJR•7/15/25, 2:40 PM

I think you might get better luck in #d1-database . I'm afraid I'm not as up to speed on the internals there.

I will say that it definitely shouldn't take 3 minutes to do 10 queries, regardless of anything else, so I'm optimistic there's a solution there.

AAJR I think you might get better luck in #d1-database . I'm afraid I'm not as up to ...

TeamDave•7/15/25, 2:40 PM

Man... I thought that's where I was, thanks!

ALECKS•7/17/25, 2:25 PM

Why doesn't the hyperdrive binding contain the connection details without having to call

.get()

.get()

? This is adds significant friction to the DX.

We're using a lot of RCP/Service workers, and it would be so natural to configure/initialize database stuff in a constructor and attach it to

this.

this.

Then every public method that could be invoked via RPC could simply use

this.db...

this.db...

. But because

.get()

.get()

is async, we need to do this dance in every public method.

AALECKS Why doesn't the hyperdrive binding contain the connection details without having...

Isaac McFadyen•7/17/25, 2:31 PM

where/how are you using

get()

get()

? Hyperdrive connection details should be on the binding itself, according to docs?

IIsaac McFadyen 🤔 where/how are you using `get()`? Hyperdrive connection details should be on t...

ALECKS•7/17/25, 2:34 PM

I'm sorry, my statement was incorrect. There's nothing wrong with the hyperdrive binding. The issue is with the secrets_store binding and I got them mixed up. Will repost in the right channel

karimulm•7/17/25, 7:10 PM

Is hyperdrive accessible from cloudflare container?

AJR•7/17/25, 9:04 PM

Not at this time, but it's something we intend to add as it's an obviously strong use case for both products.

So I have been having some sporadic issues. And I apologize for my vague "accusations" but I believe

Similar Threads

Similar Threads

Similar Threads