Oh, I think I have run into this actually.. does this translate to about 60 concurrent Workers then?
Oh, I think I have run into this actually.. does this translate to about 60 concurrent Workers then? Or more?
ASubstantially more. We see folks up past 1k QPS with this right now, glancing at logs at the moment.
ACouple reasons.
Connections do use memory and CPU and those aren't free, even for us.
Given that it's not self service to change it, if we pick a value that's too high Hyperdrive will just knock over free tier offerings from most providers. When we do make this self service we'll allow a higher maximum value to be set.
If folks hit this limit (or expect to) and reach out to us, we can set it higher manually for them for now. Many customers have. It's a good forcing function for us not to put off the self-service work for too long.
At some point we'll probably allow an absurdly high value to be set with some sort of billing implication, such that we can recoup the cost of resources if people really want to blow the doors off something. We've got some internal teams using Hyperdrive for their products set as high as 1500.
Connections do use memory and CPU and those aren't free, even for us.
Given that it's not self service to change it, if we pick a value that's too high Hyperdrive will just knock over free tier offerings from most providers. When we do make this self service we'll allow a higher maximum value to be set.
If folks hit this limit (or expect to) and reach out to us, we can set it higher manually for them for now. Many customers have. It's a good forcing function for us not to put off the self-service work for too long.
At some point we'll probably allow an absurdly high value to be set with some sort of billing implication, such that we can recoup the cost of resources if people really want to blow the doors off something. We've got some internal teams using Hyperdrive for their products set as high as 1500.
AThis one is hard for me to answer, as someone who spends more time in C or Rust than in JavaScript.
Each of those really is a separate connection, so if you do actually want to have multiple queries going at the same time than I think probably there is a difference, yes.
But on Workers I think the general design philosophy would be to do more Workers, not more stuff within a Worker, so I'm not sure how often that would come up in practice.
Each of those really is a separate connection, so if you do actually want to have multiple queries going at the same time than I think probably there is a difference, yes.
But on Workers I think the general design philosophy would be to do more Workers, not more stuff within a Worker, so I'm not sure how often that would come up in practice.
A@thomasgauvin one for your list. Not sure what info you need.
Beta isn't running yet, but should be pretty soon.
Beta isn't running yet, but should be pretty soon.
Sanyone know off the top of their head what the best way to timeout a hyperdrive query using postgres.js?
we setup the client with these opts
my understanding is that the wall clock in cf workers will only advance when there is external io performed, so setTimeout will often but not always enforce the timeout? (our metrics confirm this, but just making sure i understand why)
maybe just need to add a max_lifetime or idle_timeout to the connection options instead?
(pics are in US-mountain tz, with the cf dashboard showing avg query latency spiking around the same time as we were seeing internal metrics for max latencies higher than all the timeouts we have set in code)
we setup the client with these opts
{ connect_timeout: 1, max: 5 }setTimeout(my understanding is that the wall clock in cf workers will only advance when there is external io performed, so setTimeout will often but not always enforce the timeout? (our metrics confirm this, but just making sure i understand why)
maybe just need to add a max_lifetime or idle_timeout to the connection options instead?
(pics are in US-mountain tz, with the cf dashboard showing avg query latency spiking around the same time as we were seeing internal metrics for max latencies higher than all the timeouts we have set in code)
ASorry, not sure I'm following.
I would describe the use case of Hyperdrive as: "I, as a user, want to make a fullstack application on Workers, with my database hosted on Postgres somewhere. I will accomplish this by writing a Worker script that uses a standard Javascript Postgres driver/ORM to query my database".
Websockets can certainly play into some architectures depending on your use case, but I'd imagine most of our users do not use them.
I would describe the use case of Hyperdrive as: "I, as a user, want to make a fullstack application on Workers, with my database hosted on Postgres somewhere. I will accomplish this by writing a Worker script that uses a standard Javascript Postgres driver/ORM to query my database".
Websockets can certainly play into some architectures depending on your use case, but I'd imagine most of our users do not use them.
AIt will be handled automatically. There are two different "types" of connections at play here. The Hyperdrive->Origin connections are limited, and automatically managed. You shouldn't need to do anything to manage those except not leave a transaction open forever (which breaks Postgres anyway so probably don't do that). Whenever a query completes that connection is put back into the pool for other queries to use. The phrase to search for to learn more would be "transaction-mode connection pooler"
The Worker->Hyperdrive connections can be left open, and are not limited. That's what you'd be leaving sitting open with the architecture you're describing. That should work fine. There's a 15-second timeout on those if they're idle, so if you're going to have low traffic you may want to have a "re-open my connection" control flow path, I guess?
The Worker->Hyperdrive connections can be left open, and are not limited. That's what you'd be leaving sitting open with the architecture you're describing. That should work fine. There's a 15-second timeout on those if they're idle, so if you're going to have low traffic you may want to have a "re-open my connection" control flow path, I guess?
AThe only caveat I'd add is that a websocket connection does count as 1 of the 6 a Worker is allotted, so you'll be down 1 out of the gate and should plan your settings/query paralellism accordingly.
IIs this also the case for incoming websockets? 
IFigured that was the case for outgoing concurrent subrequest limit but didn't know it also applied to incoming
AAhhhh. I might be wrong on that. I had always assumed it was both for tcp/websockets. Better to confirm in a Workers channel . Good call, sorry about that.
II assumed it was too, but I'm not sure if it applies to incoming websockets as well as outgoing (which I think the user was referring to? Or at least I read it that way)
ASorry, missed this earlier. If it were me, I'd probably solve this on the DB side with
https://www.postgresql.org/docs/9.6/runtime-config-client.html
statement_timeouthttps://www.postgresql.org/docs/9.6/runtime-config-client.html
LHi! I discovered that Hyperdrive has not been functioning properly since around 04:00 UTC on March 10.
It seems that the connection pool is insufficient, so I increased the pool size in Supabase. However, I am still encountering a "Connection terminated unexpectedly" error. No changes have been made to the code, and I have confirmed that connecting directly to the Supabase database using the connection string from a local environment works without any issues.
Would it be possible to get some assistance regarding this issue?
It seems that the connection pool is insufficient, so I increased the pool size in Supabase. However, I am still encountering a "Connection terminated unexpectedly" error. No changes have been made to the code, and I have confirmed that connecting directly to the Supabase database using the connection string from a local environment works without any issues.
Would it be possible to get some assistance regarding this issue?
BConnected two different databases to Hyperdrive and when I try to query against them through a Worker I see it oscillate between the following two errors:
&&
Any known issues happening at the moment or is this more indicative that it's something wrong on my end?
&&
Any known issues happening at the moment or is this more indicative that it's something wrong on my end?
AWe're releasing at the moment, but that shouldn't cause either of those. Send me the IDs and I'll take a look?
AAside - those error messages should not be user-visible. This release includes some improvements to that, but if it's not better after today we'll do another pass on that. Apologies for the lack of visibility there.
KI just created a new hyperdrive connection (replacing one regional replica rds instance with another, same database spec but moving regions slightly). I’m seeing much higher/variable latencies than the previous connection (in the latency tab in metrics). I’m wondering if something about my database is slow, or if it takes hyperdrive a while to figure out where to put its pools close to my database?
Aif you DM me the IDs of new/old I can take a look in the morning
Shuh it actually ended up not being our usage of hyperdrive, nuance in the timing apis, nor our pg statement_timeout
i forgot that we were also utilizing the cf cache api tucked into this codepath, not covered by our
turns out it was the
thanks for the help regardless!
i forgot that we were also utilizing the cf cache api tucked into this codepath, not covered by our
setTimeout(turns out it was the
cache.get(thanks for the help regardless!
DI’ve definitely seen them before many times.
AOkay. Well, if you see them after this week please toss me a heads up. That is not what we want to see.
NI'm running into the error
Error: write CONNECTION_CLOSED [some value].hyperdrive.local:5432I may have actually tracked this down to just being a poor network connection between my EC2 instance and the argotunnel endpoints. I ran
cloudflared tunnel diagWhich region (and you using IPv6 or IPv4)?Just curious so I don't run into it 
NI got the error message fixed, but the diag command still seems to fail. This VPC is IPv4-only, so maybe that's why it fails? The region is us-west-2b. The other thing worth noting is that it's a dev instance, so I've tried to keep the instance size as small as will function. Ultimately, it's working reliably now by correcting an SQL query that previously had a bad cast.
AMind DM'ing me your Hyperdrive ID so I can dig into why there wasn't more info in that error message?
CHyperdrive is so cool, the private tunnel connection setup process is very slick and guided, and just works.
I saw someone mention it before, so in case anyone else comes across this: Clickhouse's Postgres interface does kind of work with Hyperdrive, I wouldn't recommend it though.
You have to enable SSL/TLS in Clickhouse config, only use Simple Queries, not extended (so no parameters, and .simple() on
Clickhouse's MySQL Interface seems to be much more fleshed out and workable, so when Hyperdrive supports that, will probably be much nicer...
I saw someone mention it before, so in case anyone else comes across this: Clickhouse's Postgres interface does kind of work with Hyperdrive, I wouldn't recommend it though.
You have to enable SSL/TLS in Clickhouse config, only use Simple Queries, not extended (so no parameters, and .simple() on
sql()Clickhouse's MySQL Interface seems to be much more fleshed out and workable, so when Hyperdrive supports that, will probably be much nicer...
VHello our product currently uses Cloudflare Workers + Neon DB + Drizzle. It's been going great but recently we're hitting scaling issues over 1M requests per day. Specifically around not enough connections.
Using Hyperdrive appeared to be the right choice and the switch seemed relatively simple.
However we've run up against a wall with this error. Queries do seem to work but i guess not all of them? We tried using
Using Hyperdrive appeared to be the right choice and the switch seemed relatively simple.
However we've run up against a wall with this error. Queries do seem to work but i guess not all of them? We tried using
fetch_types
V^ Also not sure if it's related but we do use
jsonb
KHi, I’m in @VRBaller’s team
Adding onto this, we are doing nested jsonb queries:
for example:
value->>childValue
Adding onto this, we are doing nested jsonb queries:
for example:
value->>childValue
AWe'll go ahead and bump your connection allotment in our next release. Please DM me your Hyperdrive and account IDs.
We'll be planning to cut a release starting early next week.
We'll be planning to cut a release starting early next week.
AFYI also @KingMaker
AAlso, to make sure I'm reading this right, are you having two separate issues? One is scaling, the other is a specific query?
VWe were having connection issues to our Neon Postgres. It seemed like HyperDrive was the right solution for large scale due to it's strong pooling mechanism. That might be working correctly however it seems like fetching data via Hyperdrive is behaving differently
VEssentially we're trying to go from:
to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)
to this to use Hyperdrive (it's possible this is wrong – we didn't make that many changes)
AThat is odd. That error is a type error from elsewhere in your code, but ultimately it shouldn't happen if your script had been working before and that's all you changed.
AHonestly I'm not overly familiar with neon's driver or drizzle, so I'm afraid I may not be the best positioned here.
Is the error getting thrown here when you try to connect, or later on when you go to run queries?
Is the error getting thrown here when you try to connect, or later on when you go to run queries?
AFollowup question, are you able to run any queries successfully at all?
Vqueries are showing up in hyperdrive
Vso it's connected correctly
V
VYou can see the queries showed up in Hyperdrive Analytics (while the worker was using hyperdrive)
KCc @AJR
KHi, I'm not sure if Hyperdrive is the culprit here, but I'm seeing a huge spike in errors, and I don't see any logs in my dashboard so I can't really trace it down. I did see this happen when I was having issues with Hyperdrive, so I wanted to see if I could get a look please!
Karound the same time, i see that the number of DatabaseConnections I have in RDS gets really weird, which happened the last time I had a bunch of DB disconnects
