Cloudflare Developers•6mo ago

I'm incredibly sorry to see that. Can you offer some more details about what you're seeing right now

I'm incredibly sorry to see that. Can you offer some more details about what you're seeing right now?

AJROP•7/3/25, 9:30 PM

Also, if anyone besides @Alex has some error logs coinciding with the last ~5 hours, we'd like to hear about it.

AAJR Hey @Hyperdrive ! New feature for managing connection limits just dropped: http...

Studio Smith-Cordell•7/3/25, 9:51 PM

This is great! Out of interest, what was it set to before the change?

SStudio Smith-Cordell This is great! Out of interest, what was it set to before the change?

AJROP•7/3/25, 10:00 PM

60 for paid, 20 for free

AAJR I'm incredibly sorry to see that. Can you offer some more details about what you...

Alex•7/3/25, 10:02 PM

It looks like setting lower limit and allowing a few more connection on my side resolved the issue. Currently have the limit set to 20 and allowed 35 on the MySQL user. Have not seen errors for the past ~2 hours now. So might be solved.
(and also learnt that I really need notifications when the error rate spikes like this

the notification of the changelog triggered me to look. Worker falls through to the origin though, so no biggy in this case for me)

Alex•7/3/25, 10:04 PM

That said. It would be cool to see some graphs on connection counts from the CF side and some details on what happens if I set this limit lower. Does that "just" increase query latency or will it eventually fail to execute queries if the traffic is large enough.

AJROP•7/3/25, 10:14 PM

That's in the works. We didn't want to hold up the functionality on the observability, but stay tuned.

AAlex That said. It would be cool to see some graphs on connection counts from the CF ...

AJROP•7/3/25, 10:16 PM

If you set it low enough, eventually queries will start to time out and fail, yes. We can't build up waiting queries indefinitely due to finite resources on each server, so we give them a 15 second timeout.

Alex•7/3/25, 10:18 PM

Very reasonable. Excellent!

AAJR Hey @Hyperdrive ! New feature for managing connection limits just dropped: http...

João Castro•7/4/25, 1:35 AM

Really nice!

ALECKS•7/7/25, 7:38 PM

How do you set a postgres schema when using hyperdrive? Normally I would run a

 SET search_path ..

 SET search_path ..

command but I believe that does not work... it being a session config and clients don't stay on a session.

AJROP•7/7/25, 7:47 PM

I'd recommend doing it by using

ALTER ROLE

ALTER ROLE

for the user you have set up for your Hyperdrive. That's the canonical Postgres approach and should fit well with Hyperdrive basically functioning like any other user.

AJROP•7/7/25, 7:48 PM

For my own stuff I always fully-qualify things because I have trauma from working in very-regulated-environments, but that may not be the best way to do it.

AAJR I'd recommend doing it by using `ALTER ROLE` for the user you have set up for yo...

ALECKS•7/7/25, 7:56 PM

perfect, thank you

Alex•7/7/25, 8:34 PM

So I have been having some sporadic issues. And I apologize for my vague "accusations" but I believe this might be caused by Hyperdrive or drizzle or mysql2 or some combination thereof. I have been seeing spikes of requests that receive the following error:

The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/

or this one:

Promise will never complete.

The requests timeout and fail obviously but the worker is stupid simple (it fails at 3ms CPU time and 60s wall time so it's obviously waiting for something external) and I have never seen this error on "normal" workers (those not interacting with Hyperdrive) before.

According to my Hyperdrive dashboard there were 2 errors recorded, but they don't seem to be around the time the problems happen. Also just 2 even though thousands requests failed in a short period so something somewhere is going wrong and I'm not sure where to look.

I have just tried to add timeouts everywhere I can think of but if anyone has any other idea I'm all ears

AAlex So I have been having some sporadic issues. And I apologize for my vague "accusa...

AJROP•7/7/25, 8:36 PM

When did it start? Is it still ongoing (even if sporadically)?

Alex•7/7/25, 8:42 PM

According to a client for about a week maybe 2, but honestly I am unsure. And yes this still seems to be ongoing.

AAlex According to a client for about a week maybe 2, but honestly I am unsure. And ye...

AJROP•7/7/25, 8:43 PM

Okay, I'm passing this along to my team right now, and we'll keep you posted on what we find. I'll have to reach out to the runtime team to see if we can correlate whether this is happening on other Hyperdrive-enabled workers, and what might be going on here.

If you have any other information/logs/etc that you think might be relevant please feel free to send it my way either here or via DM.

Alex•7/7/25, 8:45 PM

I'll try and send you anything I can find that might be helpful, appreciate you looking into it.

Alex•7/7/25, 8:52 PM

If we assume queries from my server take single digit ms to execute. Is it fair to assume that worldwide queries should complete within 250ms using Hyperdrive in most cases? Always exceptions ofcourse but I am trying to figure out if my timeout is too aggresive. According to the Hyperdrive dashboard latency is 31 ms past 24 hours, but unsure if that is worldwide.

AAlex If we assume queries from my server take single digit ms to execute. Is it fair ...

AJROP•7/7/25, 9:03 PM

It's so hard to say, networking and other factors make any one number I could give you either too large to be useful or likely to be wrong sometimes, I'm afraid

AAlex I'll try and send you anything I can find that might be helpful, appreciate you ...

AJROP•7/7/25, 9:04 PM

Absolutely happy to.

And please don't worry about "accusations". We are incredibly grateful to have users who are willing to go to the trouble of flagging potential problems for us. You're not going to hurt our feelings if it turns out to be something unrelated.

Isaac McFadyen•7/7/25, 9:07 PM

Out of curiosity do you specify max_connections (or whatever MySQL calls it) when creating your client?

Isaac McFadyen•7/7/25, 9:07 PM

It reminds of when I've seen this happen before with Postgres when it tries to create more than 6 connections (which fails because Workers limits TCP streams to 6 maximum)...

Isaac McFadyen•7/7/25, 9:08 PM

Which resulted in this docs issue https://github.com/cloudflare/cloudflare-docs/pull/19616 and this escalation thread (for AJR, the channel is private) https://discord.com/channels/595317990191398933/1334906924432490586/1334906924432490586

Alex•7/7/25, 9:20 PM

I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would expect this to result in a error and not the working hanging once the TCP streams limit is hit? I have since also lowered that to just 1 to be sure.

AAlex I had the default settings mysql2/drizzle set which technically cna be 10 I thin...

Isaac McFadyen•7/7/25, 9:48 PM

Theoretically the connections are queued. The issue is that new connections are queued, but old ones can potentially be held open waiting on new ones, so a deadlock occurs.

Isaac McFadyen•7/7/25, 9:48 PM

And then the runtime detects that deadlock and throws, assuming that's the issue here (it very well might not be)

Alex•7/7/25, 9:50 PM

I see, got it. So that could indeed in theory be a cause then would depend on how long the isolate lives for it to create that many connections though. My connection pool lives inside the

fetch

fetch

handler so I would expect a new "pool" every request and the old one discarded but who knows what it actually does under the hood

Alex•7/8/25, 8:13 PM

I am seeing something else strange. I am using Drizzle with mysql2 and sometimes queries return no results and also no error but are also unable to find a row. This result seems to be cached by Hyperdrive because the effects can be seen for a few seconds. I have never seen MySQL return a wrong result. I know these rows exist, and they are not modified or deleted/re-inserted or whatever. I do see a error in Hyperdrive around that time, but I don't think there is a way to correlate those errors to a specific request right now.

I run a select with limit(1) so I expect either a empty list or 1 result. So I check this with result.length === 0 and return a 404 response. However sometimes this 404 response is thus incorrect. I can see from my own logging the code thinks there is no result even though there should be one (or an error).

This might very well be a Drizzle issue but I was wondering if this might sound like something anyone has seen before or can point me in the right direction.

AJROP•7/8/25, 8:17 PM

Hmmm. We do very, very little messing with the packet structure in our mysql handler, but we deserialize it, so it's not technically impossible for that to be related to Hyperdrive.

To make sure I understand:

you send a query that's some variant of SELECT x, y, z FROM myTable LIMIT 1SELECT x, y, z FROM myTable LIMIT 1, and get a malformed response where you get 0 rows returned when you'd have expected 1 row?
this 0-length response is then cached, so you see (as you'd expect) the 0-row result persisted as cache hits for your TTL
on subsequent queries, when cache is refreshed, you get the 1 row you'd have expected

That correctly summarize what you're seeing?

Alex•7/8/25, 8:19 PM

Yes, this is exactly it. This is also intermittent. I have seen this happen at least 10 times between 18:30 and 19:30 (that I know I can filter in the logs). Not sure how many of those 10 were cached responses ofcourse but this happens from time to time.

Alex•7/8/25, 8:22 PM

I should also mention this comes in spurts. It happens a few times and then not for many hours. I was thinking this was just the uptime monitoring software acting up, but I've seen it happen a few times now and even lowered the cache TTL because of it to lessen the length of the impact.

Alex•7/8/25, 8:23 PM

If it helps I can share some request IDs, unsure that is of any use but I can make a list.

AJROP•7/8/25, 8:23 PM

Yes, a few request IDs would be useful so we can take a look

AJROP•7/8/25, 8:24 PM

Thank you

AJROP•7/8/25, 8:25 PM

I'll add a ticket to our MySQL group to take a look at this also. Appreciate you mentioning it.

Alex•7/8/25, 8:26 PM

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

95c1b0766b4a06dc
95c1b12c9ada06dc
95c1d737aed006dc
95c1d753a809e7c0
95c1d806f8de06dc
95c1f608285106dc
95c1f6105fa006dc
95c1f6ca494106dc
95c1f6dae8f206dc

Alex•7/8/25, 8:27 PM

The exact query (in case it matters):

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

select `id`, `uuid`, `team_id`, `domain`, `with_links`, `is_default`, `target`, `include_path`, `include_query`, `redirect_default_not_found` from `custom_domains` where `custom_domains`.`domain` = ? limit ?

First parameter is example.comexample.com (string) and second

(number)

AJROP•7/8/25, 8:28 PM

Got it. I would expect that that shouldn't matter, but until we have a root cause more info is always helpful.

Alex•7/10/25, 12:50 PM

FYI; this got so bad just now that I've disabled my worker for now. This sporadic nature is terrible for a links shortener as you can imagine

Giving people a 20% odds they hit a 404 instead of their intended redirect is not a good USP I have been told.

AJROP•7/10/25, 12:51 PM

Hmmm. We haven't released any changes at all to mysql handler in about a month.

AJROP•7/10/25, 12:51 PM

Okay, thanks for the heads up. That's unfortunate

Alex•7/10/25, 12:53 PM

Maybe it was always this bad and customers just started complaining but I don't know. I need to implement some checks and race my origin response vs the worker response and see how often they differ. But something is definitly broken for me.

AJROP•7/10/25, 12:55 PM

Yeah I dug through a bit yesterday and haven't been able to repro 0-length results when data is present

AJROP•7/10/25, 12:56 PM

Going to dig some more with your stack included, drizzle is popular enough that I expect it would behave well

Alex•7/10/25, 1:01 PM

I will try and see if I can figure out a way to get some useful debug output.

Alex•7/10/25, 1:02 PM

But the typeoftypeof for a succesful query was both objectobject although I know realize this was a pretty pointless thing to check for... adding some more console logs of the actual contents of the result.

Alex•7/10/25, 1:27 PM

Also slightly unrealted but not sure where to report this, the inputs on the logs page of a worker are pretty bright in dark mode (and the text is unreadable, there is something in there): https://screen.bouma.link/5xfP7KwLsj0k7wxxKXgw
Same goes for the global observability pages under the workers menu item

CleanShot Cloud

Screenshot 2025-07-10 at 15.26.22

Screenshot

AAlex But the `typeof` for a succesful query was both `object` although I know realize...

Alex•7/10/25, 6:21 PM

So as it seems it is indeed the result is incorrect without an error. I either get a array with 1 response or an empty array. It looks like it happens anywhere from 1-100 times an hour. This is from a total of 10k requests. So it's like ~1% of requests. That is pretty high actually.

AAlex So as it seems it is indeed the result is incorrect without an error. I either g...

AJROP•7/10/25, 6:23 PM

Thank you. I'm continuing to investigate, will keep you posted if I find something on our end.

I'm incredibly sorry to see that. Can you offer some more details about what you're seeing right now

Similar Threads

Similar Threads

Similar Threads