Interesting. Same connection string works in hyperdrive? What username and password did you provi

AAJR Interesting. Same connection string works in hyperdrive? What username and pa...

S

SteffTek•11/4/24, 1:19 PM

yeah sorry im just incredibly stupid. the db controller that fires first somehow still had the old d1 object as param and typescript screamed at me and i just ignored it...

A

AJROP•11/4/24, 1:30 PM

Heh. Happens to the best of us. Good luck!

D

DoggeSlapper•11/6/24, 6:59 PM

I'm trying to use hyperdrive with neon drizzle & postgresjs but I'm getting

✘ [ERROR] Uncaught (async) Error: write CONNECT_TIMEOUT 2a4a9d73ee44058fe7577f3b613e1ae9.hyperdrive.local:5432

✘ [ERROR] Uncaught (async) Error: write CONNECT_TIMEOUT 2a4a9d73ee44058fe7577f3b613e1ae9.hyperdrive.local:5432

D

DoggeSlapper•11/6/24, 7:18 PM

I been trying pg & postgresjs but cant manage to cconnect

D

DoggeSlapper•11/6/24, 7:31 PM

I have reduced to:

const sql = postgres(env.HYPERDRIVE.connectionString, { ssl: true });
const result = await sql`SELECT * FROM pg_tables;`;
console.log(result);

const sql = postgres(env.HYPERDRIVE.connectionString, { ssl: true });
const result = await sql`SELECT * FROM pg_tables;`;
console.log(result);

const sql = postgres(env.HYPERDRIVE.connectionString, { ssl: true });
const result = await sql`SELECT * FROM pg_tables;`;
console.log(result);

const sql = postgres(env.HYPERDRIVE.connectionString, { ssl: true });
const result = await sql`SELECT * FROM pg_tables;`;
console.log(result);

Without SSL true I get

✘ [ERROR] Uncaught (in promise) PostgresError: connection is insecure (try using `sslmode=require`)

✘ [ERROR] Uncaught (in promise) PostgresError: connection is insecure (try using `sslmode=require`)

A

AJROP•11/6/24, 7:31 PM

Is your database configured with

ssl=on

ssl=on

?

D

DoggeSlapper•11/6/24, 7:31 PM

When I have it enabled I get timeout error

Uncaught (in promise) Error: write CONNECT_TIMEOUT f86bad4f8437b66de3bf8efae5264dd3.hyperdrive.local:5432

Uncaught (in promise) Error: write CONNECT_TIMEOUT f86bad4f8437b66de3bf8efae5264dd3.hyperdrive.local:5432

A

AJROP•11/6/24, 7:32 PM

SSL is required to be configured for databases Hyperdrive connects to.

AAJR SSL is required to be configured for databases Hyperdrive connects to.

D

DoggeSlapper•11/6/24, 7:32 PM

it's ?sslmode=require

D

DoggeSlapper•11/6/24, 7:34 PM

I'm trying to use neon with hyperdrive from local

DDoggeSlapper it's ?sslmode=require

A

AJROP•11/6/24, 7:34 PM

That's the client-side request mode. Is the database itself set up with SSL?

AAJR That's the client-side request mode. Is the database itself set up with SSL?

D

DoggeSlapper•11/6/24, 7:39 PM

afaik neon uses ssl right?

D

DoggeSlapper•11/6/24, 7:41 PM

export WRANGLER_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE= my connection string
pnpm run dev

export WRANGLER_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE= my connection string
pnpm run dev

With ssl I get TIMEOUT

D

DoggeSlapper•11/6/24, 7:47 PM

Error: write CONNECT_TIMEOUT 00cb16054d82c846e02014158175759c.hyperdrive.local:5432
    at connectTimedOut (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
    at done (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)
    at new Query (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/query.js:35:9)
    at sql (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/index.js:113:11)
    at null.<anonymous> (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:104:27)
    at async Promise.all (index 0)
    at async Object.queue (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:97:3) {
  code: 'CONNECT_TIMEOUT',
  errno: 'CONNECT_TIMEOUT',
  address: '00cb16054d82c846e02014158175759c.hyperdrive.local',
  port: 5432,
  query: undefined,
  parameters: undefined,
  args: [],
  types: null
}

Error: write CONNECT_TIMEOUT 00cb16054d82c846e02014158175759c.hyperdrive.local:5432
    at connectTimedOut (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
    at done (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)
    at new Query (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/query.js:35:9)
    at sql (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/index.js:113:11)
    at null.<anonymous> (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:104:27)
    at async Promise.all (index 0)
    at async Object.queue (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:97:3) {
  code: 'CONNECT_TIMEOUT',
  errno: 'CONNECT_TIMEOUT',
  address: '00cb16054d82c846e02014158175759c.hyperdrive.local',
  port: 5432,
  query: undefined,
  parameters: undefined,
  args: [],
  types: null
}

Error: write CONNECT_TIMEOUT 00cb16054d82c846e02014158175759c.hyperdrive.local:5432
    at connectTimedOut (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
    at done (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)
    at new Query (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/query.js:35:9)
    at sql (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/index.js:113:11)
    at null.<anonymous> (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:104:27)
    at async Promise.all (index 0)
    at async Object.queue (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:97:3) {
  code: 'CONNECT_TIMEOUT',
  errno: 'CONNECT_TIMEOUT',
  address: '00cb16054d82c846e02014158175759c.hyperdrive.local',
  port: 5432,
  query: undefined,
  parameters: undefined,
  args: [],
  types: null
}

Error: write CONNECT_TIMEOUT 00cb16054d82c846e02014158175759c.hyperdrive.local:5432
    at connectTimedOut (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:259:20)
    at done (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/connection.js:1035:8)
    at new Query (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/query.js:35:9)
    at sql (file:///Users/leon/code/domainsHunter/node_modules/.pnpm/postgres@3.4.5/node_modules/postgres/cf/src/index.js:113:11)
    at null.<anonymous> (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:104:27)
    at async Promise.all (index 0)
    at async Object.queue (file:///Users/leon/code/domainsHunter/apps/workflows/src/index.ts:97:3) {
  code: 'CONNECT_TIMEOUT',
  errno: 'CONNECT_TIMEOUT',
  address: '00cb16054d82c846e02014158175759c.hyperdrive.local',
  port: 5432,
  query: undefined,
  parameters: undefined,
  args: [],
  types: null
}

A

AJROP•11/6/24, 7:53 PM

Ohhh, this is local dev. You're using

wrangler dev

wrangler dev

?

AAJR Ohhh, this is local dev. You're using `wrangler dev`?

D

DoggeSlapper•11/6/24, 7:54 PM

yes I'm trying to connect to neon during my local development with hyperdrive

A

AJROP•11/6/24, 7:55 PM

Got it. This is a known issue, fix is going out with the next

workerd

workerd

release.

A

AJROP•11/6/24, 7:55 PM

https://github.com/cloudflare/workerd/pull/3037

GitHub

SQC-333: Support Hyperdrive with SSL in dev mode by ReppCodes · Pul...

We have historically used sslmode=disable in when running wrangler dev, with the intention of not forcing folks to set up SSL on a local test DB. Unfortunately this causes friction because many ho...

A

AJROP•11/6/24, 7:56 PM

Sorry I don't have a better answer than that.

A

AJROP•11/6/24, 7:56 PM

Workaround is to connect to PG locally on Docker or something, so you can develop in the meantime.

D

DoggeSlapper•11/6/24, 7:56 PM

ok ok, was getting crazy on this hehehehe

D

DoggeSlapper•11/6/24, 7:57 PM

I think a got an idea to make it work locally for now

A

AJROP•11/6/24, 7:58 PM

I keep this one as an alias, I use it so much. Would need to load up your schemas too, if you have those as

.sql

.sql

files.

docker run -e POSTGRES_PASSWORD=password -p 5432:5432 postgres:16 -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

docker run -e POSTGRES_PASSWORD=password -p 5432:5432 postgres:16 -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

docker run -e POSTGRES_PASSWORD=password -p 5432:5432 postgres:16 -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

docker run -e POSTGRES_PASSWORD=password -p 5432:5432 postgres:16 -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

D

DoggeSlapper•11/6/24, 7:59 PM

https://github.com/supabase-community/pg-gateway I think think I can make it work using this as a proxy

GitHub

GitHub - supabase-community/pg-gateway: Postgres wire protocol for ...

Postgres wire protocol for the server-side. Contribute to supabase-community/pg-gateway development by creating an account on GitHub.

D

DoggeSlapper•11/6/24, 8:00 PM

but will first test with local docker postgresql

AAJR I keep this one as an alias, I use it so much. Would need to load up your schem...

I

Isaac McFadyen•11/6/24, 8:06 PM

Is a key called

snakeoil

snakeoil

actually built into the Docker container?

I

Isaac McFadyen•11/6/24, 8:06 PM

That's funny

A

AJROP•11/6/24, 8:06 PM

It's a linux-ism.
https://eleni.blog/2019/04/10/the-snakeoil-ssl-certificate/

Eleni Fragkiadakiefrag

The snakeoil ssl certificate

I recently came across this one as part of a PostgreSQL upgrade I run on my Linux (Debian) box. Going from version PostgreSQL 9.4 to 9.5 worked like a charm. Upgrading to version 9.6 did not go as …

D

DoggeSlapper•11/6/24, 8:07 PM

Thanks @AJR now with local wrangler dev without ssl works fine, excited for new release.
Loving workers so far

AAJR It's a linux-ism. https://eleni.blog/2019/04/10/the-snakeoil-ssl-certificate/

A

AJROP•11/6/24, 8:07 PM

It's been on every dev workstation I've ever used. Threw me for quite a loop when a new teammate on a Mac used my alias and it failed on them...

DDoggeSlapper Thanks @AJR now with local wrangler dev without ssl works fine, excited for new ...

A

AJROP•11/6/24, 8:08 PM

Glad to hear it! Good luck building!

N

Nick•11/6/24, 8:09 PM

My experience with Hyperdrive in production has been that it’s quite unreliable. Every so often, identical requests will encounter this error:
"Internal error." at node_modules/pg/lib/client.js:535:17

No errors are shown in the Hyperdrive Metrics Dashboard.

The documentation states that this issue is on Cloudflare's end and advises contacting support. However, with other issues I never receive a response from support, so I’m looking for an alternative solution. Any suggestions?

NNick My experience with Hyperdrive in production has been that it’s quite unreliable....

A

AJROP•11/6/24, 8:14 PM

Post/DM your Hyperdrive ID and I'll take a look. Also, can you share other info about what kinds of queries or how often you're seeing this? Is this an "every 10 queries" kind of thing, or "every 100k queries"?

Assuming these are rare failures, the guidance here would be the same as any other connection pooler you're connecting to over a network -- application-level retry logic is strongly encouraged.

AAJR It's been on every dev workstation I've ever used. Threw me for quite a loop wh...

I

Isaac McFadyen•11/6/24, 8:18 PM

Ha - interesting.

N

Noxydude•11/8/24, 4:10 PM

Hey team - I'm with UiPath Academy (academy.uipath.com) which is built on Hyperdrive (with pg) and getting 23-million weekly Hyperdrive queries. Just submitted a Limit Request Form after some of our users were getting errors during high-traffic (see second screenshot for the errors we get). Just wanted to send a message here to further troubleshoot and ensure the Limit Request Form is noticed, given this is high priority for us! Thank you.

A

AJROP•11/8/24, 4:35 PM

Thanks for bringing that to our attention. I'll discuss with the team during this morning's meeting.

A

AJROP•11/8/24, 4:38 PM

In terms of troubleshooting, some things you can do to buy yourself headroom:

Do you have transactions, especially around multiple queries, that you don't need? Those will limit how aggressively your connections get multiplexed and therefore increase your risk of pool exhaustion.

Do you have any particularly long-running queries that could be optimized on your origin DB, such as with indexes or more regular vacuuming?

A

AJROP•11/8/24, 4:41 PM

FYI @thomasgauvin ^

N

Noxydude•11/8/24, 4:44 PM

Thank you @AJR!

For SQL Inserts, Updates, Deletes, we're using transactions , doing
```
client.query('BEGIN')
```
client.query('BEGIN')
```
client.query('BEGIN')
```
client.query('BEGIN')
and
```
client.query('COMMIT")
```
client.query('COMMIT")
```
client.query('COMMIT")
```
client.query('COMMIT")
- We are getting some unrelated DB errors (e.g. pg duplicate key errors) and perhaps this is causing that Hyperdrive connection to hang? We are not calling
```
ctx.waitUntil(client.end())
```
  ctx.waitUntil(client.end())
```
ctx.waitUntil(client.end())
```
  ctx.waitUntil(client.end())
  for Hyperdrive (upon success or error) based on this old hyperdrive in May, but it looks like now the Hyperdrive docs for pg actually do recommend cleaning up the client now?
Regarding the second question - there are a few internal processes that might be long-running, we'll take a look at this.

NNoxydude Thank you @AJR! - For SQL Inserts, Updates, Deletes, we're using transactions , ...

A

AJROP•11/8/24, 4:48 PM

I'd see what you can do to trim those explicit begin/commit steps. If your application needs them then it is what it is, but they're definitely not helping with this specific issue.

Calling the waitUntil on client disconnect is a best practice and we do recommend it. It being ommitted was due to a challenge on our side that has since been resolved. It won't really help with the issue you're facing right now, though.

A

AJROP•11/8/24, 4:49 PM

I'll let you know what steps we have on the limit increase. It's not something Hyperdrive has done much yet, to be transparent, but it's sounding like it's going to start being a thing as usage increases and people scale up.

N

Noxydude•11/8/24, 4:55 PM

Great, this is super helpful @AJR. We'll add back the client disconnect, and we'll definitely test out removing the begin/commit steps to see if it helps. However, this may take a while for us to be confident in releasing, as we're currently in hypercare mode due to being just-released, so we'd definitely appreciate the limit increase

especially given our high user-base.

We understood this was a new technology when we picked it, so we definitely appreciate the effort here to help! Thank you so much!

N

Noxydude•11/8/24, 5:04 PM

@AJR One final note for additional context - we need the latest user data for almost all of our queries, so we have Hyperdrive caching disabled since according to hyperdrive it does not purge cache upon SQL updates (e.g. we'll do a SELECT, then UPDATE, then the next SELECT would be outdated)

A

AJROP•11/8/24, 5:05 PM

Understood. Obviously caching would reduce connection exhaustion, but it just isn't workable for every use case, especially without purge.

A

AJROP•11/8/24, 5:06 PM

Oh, an easier change for you would be to have a second Hyperdrive with caching enabled, and use that for any cache-tolerant queries. Not sure if that's relevant, but worth mentioning.

N

Noxydude•11/8/24, 5:09 PM

Oh, nice suggestion! We'll keep that in mind (though almost all of our queries pertain to user progress/data unfortunately)

Cchimame When will the problem with Hyperdrive in the Asian region be fixed? Previously,...

A

AJROP•11/9/24, 11:14 PM

We don't have a set timeline for it. We have a fix that has been developed, but deployment and testing is complex and ongoing. The underlying issue, intermittently poor connectivity in the APAC region, is not one that is easily solved.

When we finish rolling out our improvement I'll ping here so folks can test.

Interesting. Same connection string works in hyperdrive? What username and password did you provi

Similar Threads

Similar Threads

Similar Threads