Also, to make sure I'm reading this right, are you having two separate issues? One is scaling, the o
Also, to make sure I'm reading this right, are you having two separate issues? One is scaling, the other is a specific query?
VWe were having connection issues to our Neon Postgres. It seemed like HyperDrive was the right solution for large scale due to it's strong pooling mechanism. That might be working correctly however it seems like fetching data via Hyperdrive is behaving differently
VEssentially we're trying to go from:
to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)
to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)
AThat is odd. That error is a type error from elsewhere in your code, but ultimately it shouldn't happen if your script had been working before and that's all you changed.
AHonestly I'm not overly familiar with neon's driver or drizzle, so I'm afraid I may not be the best positioned here.
Is the error getting thrown here when you try to connect, or later on when you go to run queries?
Is the error getting thrown here when you try to connect, or later on when you go to run queries?
AFollowup question, are you able to run any queries successfully at all?
Vqueries are showing up in hyperdrive
Vso it's connected correctly
V
VYou can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)
KCc @AJR
KHi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike in errors, and I don't see any logs in my dashboard so I can't really trace it down. I did see this happen when I was having issues with Hyperdrive, so I wanted to see if I could get a look please!
Karound the same time, i see that the number of DatabaseConnections I have in RDS gets really weird, which happened the last time I had a bunch of DB disconnects
KI'm now fairly convinced that it was a Hyperdrive issue. As a hot-fix, I just deployed with a raw postgresjs client (no hyperdrive) and the errors stopped
A^ I'm experiencing similar issue, not yet deep diving to the trace but the API mostly calling to hyperdrive
about ~1 hour ago everything still good
about ~1 hour ago everything still good
Klet me know if you want to share notes on what you find
Kthis was particularly rough because i didn't see any logs in my worker that would indicate a hyperdrive issue. curious what other people were seeing
ISame here, replacing hyperdrive with another pooler solved the issue, was really hard to pinpoint
Kwhat did you use?
AHey all, can you confirm if you're still seeing impact? I'm seeing issues in some of our Virginia locations.
@kchro3 @an @patternmaster
@kchro3 @an @patternmaster
KI don’t know, I disabled Hyperdrive altogether
II'm using timescaledb and they have their own pooler
II can test tomorrow, my timescaledb is indeed in virginia
ADMing you now to collect some information
AThis was escalated as an incident, and has now been remediated. We are monitoring status to ensure things remain functioning as expected.
https://www.cloudflarestatus.com/
https://www.cloudflarestatus.com/
Welcome to Cloudflare's home for real-time and historical data on system performance.
OI am afraid we were still having issues. I created a new Hyperdrive to the same db and it works fine. CORRECTION: the new hyperdrive worked fine at first, but then Client Disconnected errors started building up again
OBtw loving the new Invocations tab in Workers Logs
AConfirmed we saw impact start climbing again. Still working through it.
AImpact has been resolved. We will monitor for a while yet.
AMonitoring is showing impact picking back up, working to remediate this.
Kwhen things have settled, can you share more about what has been causing these client disconnected issues over the past couple weeks? i was considering moving to a privately accessible database with a Hyperdrive connection, but i can't justify the risk now because at least with a publicly accessible db, i can connect to it normally.
AThings look mostly resolved, there's some backlogged traffic to work through, but if you're still seeing high error rates please let me know.
AThere's a few different things contributing to this, and picking apart which are the worst offenders requires some observability improvements that should have gone out today. I'll try to set aside some time this week to write up our plans on this topic.
Si'm now getting a few
CONNECTION_CLOSEDnot super high but i'm only sampling at 50%
SDefinitely not every invocation though, in fact it seems pretty much fine from the user side
AWell that's a good start, at least. Please send me your Hyperdrive ID so I can take a look
S9568cd870bee47f3801c862de747ca94
II'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT
YThe issue should be resolved by now. We identified the root cause and rolled out a hotfix.
Kramped traffic back up, so far ok
AHyperdrive connects to external databases (e.g Neon, RDS, on-prem Postgres, etc), so it would depend where and how you're hosting the database you're trying to connect to.
AThat makes sense. Let me explain a bit.
D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.
Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.
D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.
Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.
AIt should be less than a couple seconds.
AGetting all the way from WEUR to Bangkok might take slightly longer. If you observe longer than a minute, please let me know.
AEven a minute would be out of line.
ACache hits for Hyperdrive will be 2-7ms latency. It won't be hard to tell once it gets to working.