I'll have to go take a look when I have some more spare cycles. My teammate did the Dash work for th

I'll have to go take a look when I have some more spare cycles. My teammate did the Dash work for that one. I suspect you might be right, though.

AAJR I'll have to go take a look when I have some more spare cycles. My teammate did ...

Studio Smith-Cordell•6/26/25, 6:58 PM

Well, thank you for you and all your team's hard work. Love the CF platform

AgentCord•6/27/25, 8:18 PM

Is it better to use

pg

pg

library or

postgres

postgres

AJROP•6/27/25, 8:30 PM

pg

pg

is preferable.

AAJR Sure, I can talk through that. It's very "in the weeds" and is definitely the so...

AJROP•6/27/25, 8:31 PM

If you search in this channel you'll find discussions as to why. This is the most recent bit

Zetax•6/29/25, 6:59 PM

Will Hyperdrive support cache invalidation on writes/updates anytime soon? This is the only thing keeping me from enabling the cache on it. I'm currently caching via a custom cache built on durable objects on drizzle

ZZetax Will Hyperdrive support cache invalidation on writes/updates anytime soon? This ...

AJROP•6/29/25, 8:14 PM

No, not very soon. While it would be relatively straightforward to invalidate all cached entries on any write, getting all the way down to schema-aware automatic cache invalidation on a distributed pooler+cache is a ton of complexity that we are being very cautious about taking on.

That said, increased flexibility on whether/how to read from or write to cache on individual queries is something that we're actively working on.

AAJR No, not very soon. While it would be relatively straightforward to invalidate a...

Zetax•6/29/25, 9:31 PM

That already sounds great, thank you

Jjschlesser Sorry if this isnt exactly a hyperdrive question. It seems like the most approp...

AJROP•6/30/25, 9:21 PM

Are you planning to run with an untrusted cert when you deploy it? Or is this just a blocker for local dev?

AJROP•6/30/25, 9:35 PM

The next thing I'd try here is

npx wrangler dev --remote

npx wrangler dev --remote

, to see if you need actual code changes.

note: this will spin up an actual (ephemeral) worker for you to dev with, so be cautious about pointing your connections to a staging instance or similar

AJROP•6/30/25, 9:36 PM

database connection TLS in particular has a rough time on local dev for Workers right now, but

--remote

--remote

is usually much better behaved.

AJROP•6/30/25, 10:00 PM

Ah, I knew someone had done this before. James wrote it up publicly too, how convenient.

https://blog.cloudflare.com/introducing-socket-workers/

The Cloudflare Blog

Making connections with TCP and Sockets for Workers

The ability to make TCP and QUIC client connections from within Workers and Durable Objects, as well as the ability to connect to Workers over TCP and QUIC without using HTTP, will be coming to Cloudflare Workers. However, there is much to consider and a lot to do to make it happen. Here’s a peek at what we’re working on.

AJROP•6/30/25, 10:01 PM

Though, I won't pretend that's the easiest thing I've ever seen, either.

AJROP•7/3/25, 8:14 PM

Hey @Hyperdrive ! New feature for managing connection limits just dropped:
https://developers.cloudflare.com/changelog/2025-07-02-hyperdrive-configurable-connection-count/

Please kick the tires and let us know how it works for you!

Bonus news for those who keep an eye on this channel: as a result of this work the limit overrides that the Hyperdrive team can do to support large traffic volumes can now be done in minutes, instead of taking a few hours to roll through, so if you have need of that please reach out.

Cloudflare Docs

Hyperdrive now supports configuring the amount of database connections

Hyperdrive configurations can now be set to use a specific number of connections to your origin database.

James•7/3/25, 8:17 PM

Love to see this, thank you! :MeowHeartCloudflare:

Alex•7/3/25, 8:53 PM

I think I can see exactly when this went live

https://screen.bouma.link/XjBnMv39lxpkny2HfM3c
Love to see this change but it also changes some stuff resulting in more errors.
Already adjusting my limits but only helped for a short period.

CleanShot Cloud

Screenshot 2025-07-03 at 22.52.37

Screenshot

AJROP•7/3/25, 9:25 PM

I'm incredibly sorry to see that. Can you offer some more details about what you're seeing right now?

AJROP•7/3/25, 9:30 PM

Also, if anyone besides @Alex has some error logs coinciding with the last ~5 hours, we'd like to hear about it.

AAJR Hey @Hyperdrive ! New feature for managing connection limits just dropped: http...

Studio Smith-Cordell•7/3/25, 9:51 PM

This is great! Out of interest, what was it set to before the change?

SStudio Smith-Cordell This is great! Out of interest, what was it set to before the change?

AJROP•7/3/25, 10:00 PM

60 for paid, 20 for free

AAJR I'm incredibly sorry to see that. Can you offer some more details about what you...

Alex•7/3/25, 10:02 PM

It looks like setting lower limit and allowing a few more connection on my side resolved the issue. Currently have the limit set to 20 and allowed 35 on the MySQL user. Have not seen errors for the past ~2 hours now. So might be solved.
(and also learnt that I really need notifications when the error rate spikes like this

the notification of the changelog triggered me to look. Worker falls through to the origin though, so no biggy in this case for me)

Alex•7/3/25, 10:04 PM

That said. It would be cool to see some graphs on connection counts from the CF side and some details on what happens if I set this limit lower. Does that "just" increase query latency or will it eventually fail to execute queries if the traffic is large enough.

AJROP•7/3/25, 10:14 PM

That's in the works. We didn't want to hold up the functionality on the observability, but stay tuned.

AAlex That said. It would be cool to see some graphs on connection counts from the CF ...

AJROP•7/3/25, 10:16 PM

If you set it low enough, eventually queries will start to time out and fail, yes. We can't build up waiting queries indefinitely due to finite resources on each server, so we give them a 15 second timeout.

Alex•7/3/25, 10:18 PM

Very reasonable. Excellent!

AAJR Hey @Hyperdrive ! New feature for managing connection limits just dropped: http...

João Castro•7/4/25, 1:35 AM

Really nice!

ALECKS•7/7/25, 7:38 PM

How do you set a postgres schema when using hyperdrive? Normally I would run a

 SET search_path ..

 SET search_path ..

command but I believe that does not work... it being a session config and clients don't stay on a session.

AJROP•7/7/25, 7:47 PM

I'd recommend doing it by using

ALTER ROLE

ALTER ROLE

for the user you have set up for your Hyperdrive. That's the canonical Postgres approach and should fit well with Hyperdrive basically functioning like any other user.

AJROP•7/7/25, 7:48 PM

For my own stuff I always fully-qualify things because I have trauma from working in very-regulated-environments, but that may not be the best way to do it.

AAJR I'd recommend doing it by using `ALTER ROLE` for the user you have set up for yo...

ALECKS•7/7/25, 7:56 PM

perfect, thank you

Alex•7/7/25, 8:34 PM

So I have been having some sporadic issues. And I apologize for my vague "accusations" but I believe this might be caused by Hyperdrive or drizzle or mysql2 or some combination thereof. I have been seeing spikes of requests that receive the following error:

The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response. Refer to: https://developers.cloudflare.com/workers/observability/errors/

or this one:

Promise will never complete.

The requests timeout and fail obviously but the worker is stupid simple (it fails at 3ms CPU time and 60s wall time so it's obviously waiting for something external) and I have never seen this error on "normal" workers (those not interacting with Hyperdrive) before.

According to my Hyperdrive dashboard there were 2 errors recorded, but they don't seem to be around the time the problems happen. Also just 2 even though thousands requests failed in a short period so something somewhere is going wrong and I'm not sure where to look.

I have just tried to add timeouts everywhere I can think of but if anyone has any other idea I'm all ears

AAlex So I have been having some sporadic issues. And I apologize for my vague "accusa...

AJROP•7/7/25, 8:36 PM

When did it start? Is it still ongoing (even if sporadically)?

Alex•7/7/25, 8:42 PM

According to a client for about a week maybe 2, but honestly I am unsure. And yes this still seems to be ongoing.

AAlex According to a client for about a week maybe 2, but honestly I am unsure. And ye...

AJROP•7/7/25, 8:43 PM

Okay, I'm passing this along to my team right now, and we'll keep you posted on what we find. I'll have to reach out to the runtime team to see if we can correlate whether this is happening on other Hyperdrive-enabled workers, and what might be going on here.

If you have any other information/logs/etc that you think might be relevant please feel free to send it my way either here or via DM.

Alex•7/7/25, 8:45 PM

I'll try and send you anything I can find that might be helpful, appreciate you looking into it.

Alex•7/7/25, 8:52 PM

If we assume queries from my server take single digit ms to execute. Is it fair to assume that worldwide queries should complete within 250ms using Hyperdrive in most cases? Always exceptions ofcourse but I am trying to figure out if my timeout is too aggresive. According to the Hyperdrive dashboard latency is 31 ms past 24 hours, but unsure if that is worldwide.

AAlex If we assume queries from my server take single digit ms to execute. Is it fair ...

AJROP•7/7/25, 9:03 PM

It's so hard to say, networking and other factors make any one number I could give you either too large to be useful or likely to be wrong sometimes, I'm afraid

AAlex I'll try and send you anything I can find that might be helpful, appreciate you ...

AJROP•7/7/25, 9:04 PM

Absolutely happy to.

And please don't worry about "accusations". We are incredibly grateful to have users who are willing to go to the trouble of flagging potential problems for us. You're not going to hurt our feelings if it turns out to be something unrelated.

Isaac McFadyen•7/7/25, 9:07 PM

Out of curiosity do you specify max_connections (or whatever MySQL calls it) when creating your client?

Isaac McFadyen•7/7/25, 9:07 PM

It reminds of when I've seen this happen before with Postgres when it tries to create more than 6 connections (which fails because Workers limits TCP streams to 6 maximum)...

Isaac McFadyen•7/7/25, 9:08 PM

Which resulted in this docs issue https://github.com/cloudflare/cloudflare-docs/pull/19616 and this escalation thread (for AJR, the channel is private) https://discord.com/channels/595317990191398933/1334906924432490586/1334906924432490586

Alex•7/7/25, 9:20 PM

I had the default settings mysql2/drizzle set which technically cna be 10 I think. However I would expect this to result in a error and not the working hanging once the TCP streams limit is hit? I have since also lowered that to just 1 to be sure.

AAlex I had the default settings mysql2/drizzle set which technically cna be 10 I thin...

Isaac McFadyen•7/7/25, 9:48 PM

Theoretically the connections are queued. The issue is that new connections are queued, but old ones can potentially be held open waiting on new ones, so a deadlock occurs.

Isaac McFadyen•7/7/25, 9:48 PM

And then the runtime detects that deadlock and throws, assuming that's the issue here (it very well might not be)

Alex•7/7/25, 9:50 PM

I see, got it. So that could indeed in theory be a cause then would depend on how long the isolate lives for it to create that many connections though. My connection pool lives inside the fetchfetch handler so I would expect a new "pool" every request and the old one discarded but who knows what it actually does under the hood

Alex•7/8/25, 8:13 PM

I am seeing something else strange. I am using Drizzle with mysql2 and sometimes queries return no results and also no error but are also unable to find a row. This result seems to be cached by Hyperdrive because the effects can be seen for a few seconds. I have never seen MySQL return a wrong result. I know these rows exist, and they are not modified or deleted/re-inserted or whatever. I do see a error in Hyperdrive around that time, but I don't think there is a way to correlate those errors to a specific request right now.

I run a select with limit(1) so I expect either a empty list or 1 result. So I check this with result.length === 0 and return a 404 response. However sometimes this 404 response is thus incorrect. I can see from my own logging the code thinks there is no result even though there should be one (or an error).

This might very well be a Drizzle issue but I was wondering if this might sound like something anyone has seen before or can point me in the right direction.

I'll have to go take a look when I have some more spare cycles. My teammate did the Dash work for th

Similar Threads