My experience with Hyperdrive in production has been that it’s quite unreliable. Every so often, ide

NNick My experience with Hyperdrive in production has been that it’s quite unreliable....

A

AJR•11/6/24, 8:14 PM

Post/DM your Hyperdrive ID and I'll take a look. Also, can you share other info about what kinds of queries or how often you're seeing this? Is this an "every 10 queries" kind of thing, or "every 100k queries"?

Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.

AAJR It's been on every dev workstation I've ever used. Threw me for quite a loop wh...

I

Isaac McFadyen•11/6/24, 8:18 PM

Ha - interesting.

N

Noxydude•11/8/24, 4:10 PM

Hey team - I'm with UiPath Academy (academy.uipath.com) which is built on Hyperdrive (with pg) and getting 23-million weekly Hyperdrive queries. Just submitted a Limit Request Form after some of our users were getting errors during high-traffic (see second screenshot for the errors we get). Just wanted to send a message here to further troubleshoot and ensure the Limit Request Form is noticed, given this is high priority for us! Thank you.

A

AJR•11/8/24, 4:35 PM

Thanks for bringing that to our attention. I'll discuss with the team during this morning's meeting.

A

AJR•11/8/24, 4:38 PM

In terms of troubleshooting, some things you can do to buy yourself headroom:

Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.

Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?

A

AJR•11/8/24, 4:41 PM

FYI @thomasgauvin ^

N

Noxydude•11/8/24, 4:44 PM

Thank you @AJR!

For SQL Inserts, Updates, Deletes, we're using transactions , doing
```
client.query('BEGIN')
```
client.query('BEGIN')
```
client.query('BEGIN')
```
client.query('BEGIN')
and
```
client.query('COMMIT")
```
client.query('COMMIT")
```
client.query('COMMIT")
```
client.query('COMMIT")
- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
```
ctx.waitUntil(client.end())
```
  ctx.waitUntil(client.end())
```
ctx.waitUntil(client.end())
```
  ctx.waitUntil(client.end())
  for Hyperdrive (upon success or error) based on this old hyperdrive in May, but it looks like now the Hyperdrive docs for pg actually do recommend cleaning up the client now?
Regarding the second question - there are a few internal processes that might be long-running, we'll take a look at this.

NNoxydude Thank you @AJR! - For SQL Inserts, Updates, Deletes, we're using transactions , ...

A

AJR•11/8/24, 4:48 PM

I'd see what you can do to trim those explicit begin/commit steps. If your application needs them then it is what it is, but they're definitely not helping with this specific issue.

Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.

A

AJR•11/8/24, 4:49 PM

I'll let you know what steps we have on the limit increase. It's not something Hyperdrive has done much yet, to be transparent, but it's sounding like it's going to start being a thing as usage increases and people scale up.

N

Noxydude•11/8/24, 4:55 PM

Great, this is super helpful @AJR. We'll add back the client disconnect, and we'll definitely test out removing the begin/commit steps to see if it helps. However, this may take a while for us to be confident in releasing, as we're currently in hypercare mode due to being just-released, so we'd definitely appreciate the limit increase

especially given our high user-base.

We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!

N

Noxydude•11/8/24, 5:04 PM

@AJR One final note for additional context - we need the latest user data for almost all of our queries, so we have Hyperdrive caching disabled since according to hyperdrive it does not purge cache upon SQL updates (e.g. we'll do a SELECT, then UPDATE, then the next SELECT would be outdated)

A

AJR•11/8/24, 5:05 PM

Understood. Obviously caching would reduce connection exhaustion, but it just isn't workable for every use case, especially without purge.

A

AJR•11/8/24, 5:06 PM

Oh, an easier change for you would be to have a second Hyperdrive with caching enabled, and use that for any cache-tolerant queries. Not sure if that's relevant, but worth mentioning.

N

Noxydude•11/8/24, 5:09 PM

Oh, nice suggestion! We'll keep that in mind (though almost all of our queries pertain to user progress/data unfortunately)

Cchimame When will the problem with Hyperdrive in the Asian region be fixed? Previously,...

A

AJR•11/9/24, 11:14 PM

We don't have a set timeline for it. We have a fix that has been developed, but deployment and testing is complex and ongoing. The underlying issue, intermittently poor connectivity in the APAC region, is not one that is easily solved.

When we finish rolling out our improvement I'll ping here so folks can test.

AAJR We don't have a set timeline for it. We have a fix that has been developed, but...

Y

yevgen•11/11/24, 12:30 PM

We actually have a workaround for customers with origin databases in APAC region, if this is a case for @chimame we can place theirs specific hyperdrive config in APAC today.

Yyevgen We actually have a workaround for customers with origin databases in APAC region...

O

okku•11/11/24, 12:33 PM

I am also the one struggling this APAC networking issue, I would appreciate too if you can go for it

A

AJR•11/11/24, 12:50 PM

Eng team can with a manual process. If you have configs that need to be placed into apac, please DM me the IDs and we'll see to it.

@chimame I thought we'd worked with you on this over the summer.

A

AJR•11/11/24, 12:54 PM

Ah. Yes, the process is to place an individual config. New ones created generally won't have this applied.

I recall we might have an account level option where all your configs would go to the same locations. Let me double check

A

AJR•11/11/24, 12:55 PM

Regardless, manual process, workaround is indeed available. The systematic fix so you don't need an engineer to help with it is in progress.

B

Ben-xD•11/18/24, 11:04 PM

Before hyperdrive was added, there was an article for connecting to postgres over TCP: https://web.archive.org/web/20230519161124/https://developers.cloudflare.com/workers/databases/connect-to-postgres/. This URL now redirects to hyperdrive docs. Does anyone know why? Was the useful feature paywalled as "hyperdrive", since that also uses TCP sockets? Or is hyperdrive something better, and the old approach still works?

Connect to PostgreSQL · Cloudflare Workers docs

Cloudflare Workers supports direct connectivity to PostgreSQL instances over the Transmission Control Protocol (TCP) using the Socket API, and via …

BBen-xD Before hyperdrive was added, there was an article for connecting to postgres ove...

A

AJR•11/18/24, 11:09 PM

The second one. TCP direct connect still works, if it works for your use case.

Hyperdrive will work much better for almost everyone though.

A

AJR•11/18/24, 11:11 PM

Also, as an interesting side note, Hyperdrive doesn't actually use TCP sockets for almost anything. I think only local dev might be an exception?

B

Ben-xD•11/18/24, 11:14 PM

Thanks!

T

thomasgauvin•11/19/24, 4:01 PM

Hyperdrive update:

2024-11-19: Hyperdrive now supports clear-text password authentication. When your database requires your password in clear-text, Hyperdrive will send the required credentials securely over TLS during the connection setup.

https://developers.cloudflare.com/hyperdrive/platform/changelog/#2024-11-19

A

alex (he/him)•11/20/24, 3:11 PM

Is it possible to use

localConnectionString

localConnectionString

localConnectionString

localConnectionString with a remote connectionString, we use Neon for development, so every developer would need to set it's own connectionString, we where thinking of using

localConnectionString

localConnectionString

localConnectionString

localConnectionString for this, however it always timesout

A

Agung Vivekananda•11/20/24, 3:15 PM

Why sometimes i got this error
message:

"Database error: PostgresError: Protocol Error: Unable to close prepared statement: hesvlatshn72",

(I use hyperdrive -> Supabase and CF worker backend)

Aalex (he/him)Is it possible to use `localConnectionString` with a remote connectionString, we...

A

AJR•11/20/24, 3:18 PM

It's meant to be possible, yes, as of a couple weeks ago. I'm getting those timeouts too. I'll open a ticket to take a look when we can. Thanks!

AAgung Vivekananda Why sometimes i got this error message: "Database error: PostgresError: Proto...

A

AJR•11/20/24, 3:19 PM

Which driver and/or ORM are you using?

T

thomasgauvin•11/20/24, 3:19 PM

@alex (he/him) what version of Wrangler are you using? it works on the latest 3.88.0

A

AJR•11/20/24, 3:20 PM

I hit it with Neon just now on 3.88.0

T

thomasgauvin•11/20/24, 3:20 PM

And @alex (he/him) do you have IP allowlisting for each developers' IP address? Might be responsible for timeout

A

alex (he/him)•11/20/24, 3:22 PM

@thomasgauvin I'm using latest 3.88.0, and I have allowed connection from

0.0.0.0/0

0.0.0.0/0

, so I really don't think it's the cause here

A

alex (he/him)•11/20/24, 3:23 PM

It also happens for RDS & Neon, so two different provider.. If there is anything else I can provide, logs? Or you want me to run any command, let me know. I can provide those.

T

thomasgauvin•11/20/24, 3:24 PM

And presumably, developers can sign in from local machine to the pg database directly in Node or via PgAdmin?

A

alex (he/him)•11/20/24, 3:24 PM

Yes that's correct! However using the "original" connectionString, I have not tried to use the hyperdrive specific one

postgresql://XXX:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/database?sslmode=prefer

postgresql://XXX:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/database?sslmode=prefer

postgresql://XXX:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/database?sslmode=prefer

postgresql://XXX:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/database?sslmode=prefer

A

AJR•11/20/24, 3:25 PM

The Hyperdrive one won't work outside a Worker, that's an internally-generated identifier

A

AJR•11/20/24, 3:26 PM

If you deploy a random test worker with your configs it behaves as expected?

A

AJR•11/20/24, 3:26 PM

Any particular SSL/TLS configuration you've set up on your origin DBs? I assume not, just checking

A

alex (he/him)•11/20/24, 3:26 PM

Production works as expected. It really is about

localConnectionString

localConnectionString

localConnectionString

localConnectionString combined with a db not running on localhost.

A

alex (he/him)•11/20/24, 3:27 PM

No particular SSL/TLS.. I would need to confirm if

localConnectionString

localConnectionString

localConnectionString

localConnectionString works with a db on localhost

A

alex (he/him)•11/20/24, 3:28 PM

{"DB":{"connectionString":"postgresql://repguide_owner:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/repguide?sslmode=prefer","port":5432,"host":"ffca328615404666884948d7d21ccf36.hyperdrive.local","password":"XXX","user":"repguide_owner","database":"repguide"}}
✘ [ERROR] Uncaught (in promise) Error: write CONNECT_TIMEOUT ffca328615404666884948d7d21ccf36.hyperdrive.local:5432

      at connectTimedOut
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
      at done
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)

{"DB":{"connectionString":"postgresql://repguide_owner:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/repguide?sslmode=prefer","port":5432,"host":"ffca328615404666884948d7d21ccf36.hyperdrive.local","password":"XXX","user":"repguide_owner","database":"repguide"}}
✘ [ERROR] Uncaught (in promise) Error: write CONNECT_TIMEOUT ffca328615404666884948d7d21ccf36.hyperdrive.local:5432

      at connectTimedOut
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
      at done
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)

{"DB":{"connectionString":"postgresql://repguide_owner:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/repguide?sslmode=prefer","port":5432,"host":"ffca328615404666884948d7d21ccf36.hyperdrive.local","password":"XXX","user":"repguide_owner","database":"repguide"}}
✘ [ERROR] Uncaught (in promise) Error: write CONNECT_TIMEOUT ffca328615404666884948d7d21ccf36.hyperdrive.local:5432

      at connectTimedOut
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
      at done
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)

{"DB":{"connectionString":"postgresql://repguide_owner:XXX@ffca328615404666884948d7d21ccf36.hyperdrive.local:5432/repguide?sslmode=prefer","port":5432,"host":"ffca328615404666884948d7d21ccf36.hyperdrive.local","password":"XXX","user":"repguide_owner","database":"repguide"}}
✘ [ERROR] Uncaught (in promise) Error: write CONNECT_TIMEOUT ffca328615404666884948d7d21ccf36.hyperdrive.local:5432

      at connectTimedOut
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
      at done
  (file:///Users/alexanderniebuhr/Developer/Projects/frickegroup/workers/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)

A

AJR•11/20/24, 3:28 PM

Yep. Ok, I'll circle up with Thomas to dig into this when we have some spare cycles. Making this work with remotes is a new feature, and it's worked for us in our testing, but it seems there's some configuration still to be done on our side.

A

AJR•11/20/24, 3:29 PM

Thanks for letting us know

My experience with Hyperdrive in production has been that it’s quite unreliable. Every so often, ide

Similar Threads

Similar Threads

Similar Threads