You can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)
You can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)
KCc @AJR
KHi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike in errors, and I don't see any logs in my dashboard so I can't really trace it down. I did see this happen when I was having issues with Hyperdrive, so I wanted to see if I could get a look please!
Karound the same time, i see that the number of DatabaseConnections I have in RDS gets really weird, which happened the last time I had a bunch of DB disconnects
KI'm now fairly convinced that it was a Hyperdrive issue. As a hot-fix, I just deployed with a raw postgresjs client (no hyperdrive) and the errors stopped
A^ I'm experiencing similar issue, not yet deep diving to the trace but the API mostly calling to hyperdrive
about ~1 hour ago everything still good
about ~1 hour ago everything still good
Klet me know if you want to share notes on what you find
Kthis was particularly rough because i didn't see any logs in my worker that would indicate a hyperdrive issue. curious what other people were seeing
ISame here, replacing hyperdrive with another pooler solved the issue, was really hard to pinpoint
Kwhat did you use?
AHey all, can you confirm if you're still seeing impact? I'm seeing issues in some of our Virginia locations.
@kchro3 @an @patternmaster
@kchro3 @an @patternmaster
KI don’t know, I disabled Hyperdrive altogether
II'm using timescaledb and they have their own pooler
II can test tomorrow, my timescaledb is indeed in virginia
ADMing you now to collect some information
AThis was escalated as an incident, and has now been remediated. We are monitoring status to ensure things remain functioning as expected.
https://www.cloudflarestatus.com/
https://www.cloudflarestatus.com/
Welcome to Cloudflare's home for real-time and historical data on system performance.
OI am afraid we were still having issues. I created a new Hyperdrive to the same db and it works fine. CORRECTION: the new hyperdrive worked fine at first, but then Client Disconnected errors started building up again
OBtw loving the new Invocations tab in Workers Logs
AConfirmed we saw impact start climbing again. Still working through it.
AImpact has been resolved. We will monitor for a while yet.
AMonitoring is showing impact picking back up, working to remediate this.
Kwhen things have settled, can you share more about what has been causing these client disconnected issues over the past couple weeks? i was considering moving to a privately accessible database with a Hyperdrive connection, but i can't justify the risk now because at least with a publicly accessible db, i can connect to it normally.
AThings look mostly resolved, there's some backlogged traffic to work through, but if you're still seeing high error rates please let me know.
AThere's a few different things contributing to this, and picking apart which are the worst offenders requires some observability improvements that should have gone out today. I'll try to set aside some time this week to write up our plans on this topic.
Si'm now getting a few
CONNECTION_CLOSEDnot super high but i'm only sampling at 50%
SDefinitely not every invocation though, in fact it seems pretty much fine from the user side
AWell that's a good start, at least. Please send me your Hyperdrive ID so I can take a look
S9568cd870bee47f3801c862de747ca94
II'm still seeing the same, most connections hang and end up in CONNECT_TIMEOUT
YThe issue should be resolved by now. We identified the root cause and rolled out a hotfix.
Kramped traffic back up, so far ok
AHyperdrive connects to external databases (e.g Neon, RDS, on-prem Postgres, etc), so it would depend where and how you're hosting the database you're trying to connect to.
AThat makes sense. Let me explain a bit.
D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.
Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.
D1 is a Cloudflare-native data storage product. It is a truly distributed product, and runs on our own network with our own tech stack. This means it will (usually) be more performant. However it does have some pretty sharp restrictions to its use, such as size or features, when compared to a more "standard" Postgres database.
Sometimes those restrictions will render D1 inappropriate for some use cases. In those cases, it would be nice to have the option to run your applications on a more standard Postgres offering either on-prem or from other vendors. In that case we still want people to be able to use Cloudflare's Developer Platform to build their application, so we built Hyperdrive. It is a tool for connecting from Workers to those other vendors' products. Because those are not on our network or our servers, connecting to them involves a network hop to wherever they're running. This means that Hyperdrive will often not be able to compete on raw latency alone. However, if you need some features that are not available on D1/R2/KV etc, now you have a way to access them.
AIt should be less than a couple seconds.
AGetting all the way from WEUR to Bangkok might take slightly longer. If you observe longer than a minute, please let me know.
AEven a minute would be out of line.
ACache hits for Hyperdrive will be 2-7ms latency. It won't be hard to tell once it gets to working.
AHappy to. Good luck!
TLet us know how it goes @PatrickJ! Also, depending on how cache heavy your workloads are you may find workers smart placement interesting. If you have more cold reads/less cache hits and a lot of subrequests per worker invocation, it may be best to have that worker placed near your data source
https://developers.cloudflare.com/workers/configuration/smart-placement/
https://developers.cloudflare.com/workers/configuration/smart-placement/
Cloudflare Docs
Speed up your Worker application by automatically placing your workloads in an optimal location that minimizes latency.
AIt'll take some traffic to kick on, but what it'll do when run with Hyperdrive is eventually learn to run the Worker itself right next to your DB. That'll make cache misses way faster, but slow down cache hits. Whether that's an improvement will depend on how often you get cache hits, like Thomas said.
AThat one I'm not sure on, sorry.
KOur Cloudflare Worker (backed by Hyperdrive) had a big spike in Errors and Wall Time today starting at around 10:30am PT today. On Hyperdrive, I don't see any spikes in latency, but I did see a couple errors in each of our Hyperdrive instances at ~10:30am PT. Struggling a little with how to debug or fix this - most traffic is fine, but our P999 wall time jumped to 70k ms. All of our backing databases are completely normal, and I'm able to query Hyperdrive normally locally
EYes in the context! Something like this middleware is recommended
Note that I'm wrapping the sql connection in another class for mocking purposes
Note that I'm wrapping the sql connection in another class for mocking purposes
IIs there a need to create+close the connection on every request here, given that Hyperdrive does pooling and so it wouldn't exhaust connection limits?
DI'd also add a maximum pool size to the postgres<->hyperdrive part, since >6 queries can deadlock the worker
ABlog coming soon. TLDR: that's a different "kind" of connection than the ones to the origin, takes ~2ms to open, and closing it will avoid holding your Worker open doing any processing, helping keep your CPU time down. I'd recommend doing it the way Emilio showed.
