Thanks @AJR now with local wrangler dev without ssl works fine, excited for new release. Loving work
Thanks @AJR now with local wrangler dev without ssl works fine, excited for new release.
Loving workers so far
Loving workers so far
AIt's been on every dev workstation I've ever used. Threw me for quite a loop when a new teammate on a Mac used my alias and it failed on them...
AGlad to hear it! Good luck building!
NMy experience with Hyperdrive in production has been that it’s quite unreliable. Every so often, identical requests will encounter this error:
"Internal error." at node_modules/pg/lib/client.js:535:17
No errors are shown in the Hyperdrive Metrics Dashboard.
The documentation states that this issue is on Cloudflare's end and advises contacting support. However, with other issues I never receive a response from support, so I’m looking for an alternative solution. Any suggestions?
"Internal error." at node_modules/pg/lib/client.js:535:17
No errors are shown in the Hyperdrive Metrics Dashboard.
The documentation states that this issue is on Cloudflare's end and advises contacting support. However, with other issues I never receive a response from support, so I’m looking for an alternative solution. Any suggestions?
APost/DM your Hyperdrive ID and I'll take a look. Also, can you share other info about what kinds of queries or how often you're seeing this? Is this an "every 10 queries" kind of thing, or "every 100k queries"?
Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.
Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.
IHa - interesting.
NHey team - I'm with UiPath Academy (academy.uipath.com) which is built on Hyperdrive (with pg) and getting 23-million weekly Hyperdrive queries. Just submitted a Limit Request Form after some of our users were getting errors during high-traffic (see second screenshot for the errors we get). Just wanted to send a message here to further troubleshoot and ensure the Limit Request Form is noticed, given this is high priority for us! Thank you.
AThanks for bringing that to our attention. I'll discuss with the team during this morning's meeting.
AIn terms of troubleshooting, some things you can do to buy yourself headroom:
Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.
Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?
Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.
Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?
AFYI @thomasgauvin ^
NThank you @AJR!
- For SQL Inserts, Updates, Deletes, we're using transactions , doing
client.query('BEGIN')andclient.query('COMMIT")- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
ctx.waitUntil(client.end())for Hyperdrive (upon success or error) based on this old hyperdrive in May, but it looks like now the Hyperdrive docs for pg actually do recommend cleaning up the client now?
- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
- Regarding the second question - there are a few internal processes that might be long-running, we'll take a look at this.
AI'd see what you can do to trim those explicit begin/commit steps. If your application needs them then it is what it is, but they're definitely not helping with this specific issue.
Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.
Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.
AI'll let you know what steps we have on the limit increase. It's not something Hyperdrive has done much yet, to be transparent, but it's sounding like it's going to start being a thing as usage increases and people scale up.
NGreat, this is super helpful @AJR. We'll add back the client disconnect, and we'll definitely test out removing the begin/commit steps to see if it helps. However, this may take a while for us to be confident in releasing, as we're currently in hypercare mode due to being just-released, so we'd definitely appreciate the limit increase 
especially given our high user-base.
We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!
especially given our high user-base.We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!
N@AJR One final note for additional context - we need the latest user data for almost all of our queries, so we have Hyperdrive caching disabled since according to hyperdrive it does not purge cache upon SQL updates (e.g. we'll do a SELECT, then UPDATE, then the next SELECT would be outdated)
AUnderstood. Obviously caching would reduce connection exhaustion, but it just isn't workable for every use case, especially without purge.
AOh, an easier change for you would be to have a second Hyperdrive with caching enabled, and use that for any cache-tolerant queries. Not sure if that's relevant, but worth mentioning.
NOh, nice suggestion! We'll keep that in mind (though almost all of our queries pertain to user progress/data unfortunately)
AWe don't have a set timeline for it. We have a fix that has been developed, but deployment and testing is complex and ongoing. The underlying issue, intermittently poor connectivity in the APAC region, is not one that is easily solved.
When we finish rolling out our improvement I'll ping here so folks can test.
When we finish rolling out our improvement I'll ping here so folks can test.
YWe actually have a workaround for customers with origin databases in APAC region, if this is a case for @chimame we can place theirs specific hyperdrive config in APAC today.
OI am also the one struggling this APAC networking issue, I would appreciate too if you can go for it
AEng team can with a manual process. If you have configs that need to be placed into apac, please DM me the IDs and we'll see to it.
@chimame I thought we'd worked with you on this over the summer.
@chimame I thought we'd worked with you on this over the summer.
AAh. Yes, the process is to place an individual config. New ones created generally won't have this applied.
I recall we might have an account level option where all your configs would go to the same locations. Let me double check
I recall we might have an account level option where all your configs would go to the same locations. Let me double check
ARegardless, manual process, workaround is indeed available. The systematic fix so you don't need an engineer to help with it is in progress.
BBefore hyperdrive was added, there was an article for connecting to postgres over TCP: https://web.archive.org/web/20230519161124/https://developers.cloudflare.com/workers/databases/connect-to-postgres/. This URL now redirects to hyperdrive docs. Does anyone know why? Was the useful feature paywalled as "hyperdrive", since that also uses TCP sockets? Or is hyperdrive something better, and the old approach still works?
Cloudflare Workers supports direct connectivity to PostgreSQL instances over the Transmission Control Protocol (TCP) using the Socket API, and via …
AThe second one. TCP direct connect still works, if it works for your use case.
Hyperdrive will work much better for almost everyone though.
Hyperdrive will work much better for almost everyone though.
AAlso, as an interesting side note, Hyperdrive doesn't actually use TCP sockets for almost anything. I think only local dev might be an exception?
BThanks!
T
Hyperdrive update: 2024-11-19: Hyperdrive now supports clear-text password authentication. When your database requires your password in clear-text, Hyperdrive will send the required credentials securely over TLS during the connection setup.
https://developers.cloudflare.com/hyperdrive/platform/changelog/#2024-11-19
AIs it possible to use 
localConnectionString with a remote connectionString, we use Neon for development, so every developer would need to set it's own connectionString, we where thinking of using localConnectionString for this, however it always timesout 
AWhy sometimes i got this error
message:
"Database error: PostgresError: Protocol Error: Unable to close prepared statement: hesvlatshn72",
(I use hyperdrive -> Supabase and CF worker backend)
message:
"Database error: PostgresError: Protocol Error: Unable to close prepared statement: hesvlatshn72",
(I use hyperdrive -> Supabase and CF worker backend)
AIt's meant to be possible, yes, as of a couple weeks ago. I'm getting those timeouts too. I'll open a ticket to take a look when we can. Thanks!
AWhich driver and/or ORM are you using?
T@alex (he/him) what version of Wrangler are you using? it works on the latest 3.88.0
AI hit it with Neon just now on 3.88.0
TAnd @alex (he/him) do you have IP allowlisting for each developers' IP address? Might be responsible for timeout
A@thomasgauvin I'm using latest 3.88.0, and I have allowed connection from
0.0.0.0/0It also happens for RDS & Neon, so two different provider.. If there is anything else I can provide, logs? Or you want me to run any command, let me know. I can provide those.
TAnd presumably, developers can sign in from local machine to the pg database directly in Node or via PgAdmin?
AYes that's correct! However using the "original" connectionString, I have not tried to use the hyperdrive specific one
postgresql://XXX:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/database?sslmode=preferAThe Hyperdrive one won't work outside a Worker, that's an internally-generated identifier
AIf you deploy a random test worker with your configs it behaves as expected?
AAny particular SSL/TLS configuration you've set up on your origin DBs? I assume not, just checking
AProduction works as expected. It really is about
localConnectionString combined with a db not running on localhost.ANo particular SSL/TLS.. I would need to confirm if
localConnectionString works with a db on localhost