That's the client-side request mode. Is the database itself set up with SSL?
That's the client-side request mode. Is the database itself set up with SSL?
Dafaik neon uses ssl right?
DWith ssl I get TIMEOUT
D
AOhhh, this is local dev. You're using
wrangler devyes I'm trying to connect to neon during my local development with hyperdrive
AGot it. This is a known issue, fix is going out with the next
workerd release.AGitHub
We have historically used sslmode=disable in when running wrangler dev, with the intention of not forcing folks to set up SSL on a local test DB. Unfortunately this causes friction because many ho...
ASorry I don't have a better answer than that.
AWorkaround is to connect to PG locally on Docker or something, so you can develop in the meantime.
Dok ok, was getting crazy on this hehehehe
DI think a got an idea to make it work locally for now
AI keep this one as an alias, I use it so much. Would need to load up your schemas too, if you have those as
.sql files.docker run -e POSTGRES_PASSWORD=password -p 5432:5432 postgres:16 -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.keyDhttps://github.com/supabase-community/pg-gateway I think think I can make it work using this as a proxy
GitHub
Postgres wire protocol for the server-side. Contribute to supabase-community/pg-gateway development by creating an account on GitHub.
Dbut will first test with local docker postgresql
IIs a key called 
snakeoil actually built into the Docker container? IThat's funny
AEleni Fragkiadakiefrag
I recently came across this one as part of a PostgreSQL upgrade I run on my Linux (Debian) box. Going from version PostgreSQL 9.4 to 9.5 worked like a charm. Upgrading to version 9.6 did not go as …
DThanks @AJR now with local wrangler dev without ssl works fine, excited for new release.
Loving workers so far
Loving workers so far
AIt's been on every dev workstation I've ever used. Threw me for quite a loop when a new teammate on a Mac used my alias and it failed on them...
AGlad to hear it! Good luck building!
NMy experience with Hyperdrive in production has been that it’s quite unreliable. Every so often, identical requests will encounter this error:
"Internal error." at node_modules/pg/lib/client.js:535:17
No errors are shown in the Hyperdrive Metrics Dashboard.
The documentation states that this issue is on Cloudflare's end and advises contacting support. However, with other issues I never receive a response from support, so I’m looking for an alternative solution. Any suggestions?
"Internal error." at node_modules/pg/lib/client.js:535:17
No errors are shown in the Hyperdrive Metrics Dashboard.
The documentation states that this issue is on Cloudflare's end and advises contacting support. However, with other issues I never receive a response from support, so I’m looking for an alternative solution. Any suggestions?
APost/DM your Hyperdrive ID and I'll take a look. Also, can you share other info about what kinds of queries or how often you're seeing this? Is this an "every 10 queries" kind of thing, or "every 100k queries"?
Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.
Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.
IHa - interesting.
NHey team - I'm with UiPath Academy (academy.uipath.com) which is built on Hyperdrive (with pg) and getting 23-million weekly Hyperdrive queries. Just submitted a Limit Request Form after some of our users were getting errors during high-traffic (see second screenshot for the errors we get). Just wanted to send a message here to further troubleshoot and ensure the Limit Request Form is noticed, given this is high priority for us! Thank you.
AThanks for bringing that to our attention. I'll discuss with the team during this morning's meeting.
AIn terms of troubleshooting, some things you can do to buy yourself headroom:
Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.
Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?
Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.
Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?
AFYI @thomasgauvin ^
NThank you @AJR!
- For SQL Inserts, Updates, Deletes, we're using transactions , doing
andclient.query('BEGIN')client.query('COMMIT")- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
for Hyperdrive (upon success or error) based on this old hyperdrive in May, but it looks like now the Hyperdrive docs for pg actually do recommend cleaning up the client now?ctx.waitUntil(client.end())
- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
- Regarding the second question - there are a few internal processes that might be long-running, we'll take a look at this.
AI'd see what you can do to trim those explicit begin/commit steps. If your application needs them then it is what it is, but they're definitely not helping with this specific issue.
Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.
Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.
AI'll let you know what steps we have on the limit increase. It's not something Hyperdrive has done much yet, to be transparent, but it's sounding like it's going to start being a thing as usage increases and people scale up.
NGreat, this is super helpful @AJR. We'll add back the client disconnect, and we'll definitely test out removing the begin/commit steps to see if it helps. However, this may take a while for us to be confident in releasing, as we're currently in hypercare mode due to being just-released, so we'd definitely appreciate the limit increase 
especially given our high user-base.
We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!
especially given our high user-base.We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!
N@AJR One final note for additional context - we need the latest user data for almost all of our queries, so we have Hyperdrive caching disabled since according to hyperdrive it does not purge cache upon SQL updates (e.g. we'll do a SELECT, then UPDATE, then the next SELECT would be outdated)
AUnderstood. Obviously caching would reduce connection exhaustion, but it just isn't workable for every use case, especially without purge.
AOh, an easier change for you would be to have a second Hyperdrive with caching enabled, and use that for any cache-tolerant queries. Not sure if that's relevant, but worth mentioning.
NOh, nice suggestion! We'll keep that in mind (though almost all of our queries pertain to user progress/data unfortunately)
AWe don't have a set timeline for it. We have a fix that has been developed, but deployment and testing is complex and ongoing. The underlying issue, intermittently poor connectivity in the APAC region, is not one that is easily solved.
When we finish rolling out our improvement I'll ping here so folks can test.
When we finish rolling out our improvement I'll ping here so folks can test.
YWe actually have a workaround for customers with origin databases in APAC region, if this is a case for @chimame we can place theirs specific hyperdrive config in APAC today.
OI am also the one struggling this APAC networking issue, I would appreciate too if you can go for it
AEng team can with a manual process. If you have configs that need to be placed into apac, please DM me the IDs and we'll see to it.
@chimame I thought we'd worked with you on this over the summer.
@chimame I thought we'd worked with you on this over the summer.
AAh. Yes, the process is to place an individual config. New ones created generally won't have this applied.
I recall we might have an account level option where all your configs would go to the same locations. Let me double check
I recall we might have an account level option where all your configs would go to the same locations. Let me double check
ARegardless, manual process, workaround is indeed available. The systematic fix so you don't need an engineer to help with it is in progress.
BBefore hyperdrive was added, there was an article for connecting to postgres over TCP: https://web.archive.org/web/20230519161124/https://developers.cloudflare.com/workers/databases/connect-to-postgres/. This URL now redirects to hyperdrive docs. Does anyone know why? Was the useful feature paywalled as "hyperdrive", since that also uses TCP sockets? Or is hyperdrive something better, and the old approach still works?
Cloudflare Workers supports direct connectivity to PostgreSQL instances over the Transmission Control Protocol (TCP) using the Socket API, and via …
AThe second one. TCP direct connect still works, if it works for your use case.
Hyperdrive will work much better for almost everyone though.
Hyperdrive will work much better for almost everyone though.
AAlso, as an interesting side note, Hyperdrive doesn't actually use TCP sockets for almost anything. I think only local dev might be an exception?
