Hyperdrive doesn’t accept arbitrary client connections itself today. It’s designed to help folks use

Hyperdrive doesn’t accept arbitrary client connections itself today. It’s designed to help folks use Workers to connect to their databases (including Supabase or Neon!).

Supabase.js makes HTTP requests directly to Supabase’s custom HTTP proxy (likely using pgrest in some way) as well - it’s not something we can really support given it’s very Supabase specific.

Mas Inis•11/19/23, 5:29 AM

Hi folks, what would be good managed postgre candidate to use with hyperdrive? (Or workers in general)

Mas Inis•11/19/23, 5:32 AM

I see Neon and Supabase mentioned quite often

MMas Inis Hi folks, what would be good managed postgre candidate to use with hyperdrive? (...

Colin•11/19/23, 3:37 PM

Just throwing in my 2c, we evaluated all of the popular “serverless” Postgres DBs and none of them ticked all the boxes/gave us as much confidence as AWS Aurora

Colin•11/19/23, 3:38 PM

I feel like Neon will be a great choice in a year or two, but rn it seemed to lack some basic features that AWS has

Isaac McFadyen•11/20/23, 2:31 PM

Google Cloud SQL is also quite good if you're already integrated into their ecosystem.

arbfay•11/21/23, 11:10 AM

I know more databases are coming later, but may I still ask if ClickHouse is on your list?

Aarbfay I know more databases are coming later, but may I still ask if ClickHouse is on ...

Matt SilverlockOP•11/21/23, 12:46 PM

Not yet, no. The ClickHouse protocol doesn’t have a specification: https://clickhouse.com/docs/en/interfaces/tcp

Native Interface (TCP) | ClickHouse Docs

The native protocol is used in the command-line client, for inter-server communication during distributed query processing, and also in other C++ programs. Unfortunately, native ClickHouse protocol does not have formal specification yet, but it can be reverse-engineered from ClickHouse source code (starting around here) and/or by intercepting an...

Aarbfay I know more databases are coming later, but may I still ask if ClickHouse is on ...

Matt SilverlockOP•11/21/23, 12:48 PM

Also: are you querying ClickHouse on every request? I can’t imagine that will be fast regardless.

MMatt Silverlock Also: are you querying ClickHouse on every request? I can’t imagine that will be...

DaniFoldi•11/21/23, 12:53 PM

With hyperdrive’s caching that could become possible meowwink

DDaniFoldi With hyperdrive’s caching that could become possible <:meowwink:3804724056778342...

Matt SilverlockOP•11/21/23, 1:20 PM

The caching relies on the protocol being specified (and a stable parser being available, plus a JS driver). But I’m asking as if you’re instead just querying it periodically/in a Cron/etc, the HTTP interface for ClickHouse is likely a more straightforward approach right now.

MMatt Silverlock Not yet, no. The ClickHouse protocol doesn’t have a specification: https://click...

arbfay•11/21/23, 2:26 PM

Ah I didn't know. I guess an experiment with their PostgreSQL protocol is necessary

MMatt Silverlock Also: are you querying ClickHouse on every request? I can’t imagine that will be...

arbfay•11/21/23, 2:28 PM

Sry I'm not sure I see why you think this is a problem.
ClickHouse for OLAP is very performant. I use it for about 600 dashboards (for different users), and yes queries every time (but cached with a worker). It's also what powers Cloudflare's Analytics Engine IIRC.

arbfay•11/21/23, 2:29 PM

It's got an HTTP endpoint to run queries, and honestly I might just stick to the Worker for caching + HTTP. I guess the most interesting part for me was the connection pooling.

Aarbfay Sry I'm not sure I see why you think this is a problem. ClickHouse for OLAP is v...

Matt SilverlockOP•11/21/23, 2:43 PM

I am very aware of how ClickHouse performs. What I am trying to point out is that if you rely on native ClickHouse support you may be waiting a long time, and that CH’s baseline performance may be “fast” but isn’t commonly sub-second for anything but the most simple queries.

Instead, using a HTTP interface will likely get you moving in the meantime, or trying their Postgres interface (untested).

MMatt Silverlock I am very aware of how ClickHouse performs. What I am trying to point out is tha...

arbfay•11/21/23, 3:06 PM

Yeah, I use their HTTP interface since Workers didn't have TCP support yet when we decided to use it and I couldn't be bothered with creating and maintaining a HTTP proxy for any other OLAP/time-series DB. Performance-wise, the queries seem to run fast enough (20-200ms, we really aren't the same size as CF), helped with caching, but the latency is spiky (I suspect its due to making new the connections but haven't investigated yet).

arbfay•11/21/23, 3:06 PM

Anyway, it's all good to know. Might try the Postgres interface some time later

Aarbfay Yeah, I use their HTTP interface since Workers didn't have TCP support yet when ...

Cyb3r-Jak3•11/21/23, 3:28 PM

Workers have outbound TCP connections.

CCyb3r-Jak3 Workers have outbound TCP connections.

arbfay•11/21/23, 3:33 PM

Now, yes. Not last year.

MMatt Silverlock Hyperdrive doesn’t accept arbitrary client connections itself today. It’s design...

DanielZero•11/25/23, 10:56 AM

It's not so supabase specific, actually. One of their core principles is using existing or their own open source software. In this case it's https://github.com/PostgREST/postgrest

DDanielZero It's not so supabase specific, actually. One of their core principles is using e...

Matt SilverlockOP•11/25/23, 10:57 AM

I mention pgrest in my comment.

DanielZero•11/25/23, 10:59 AM

Yeah I know, just wanted to throw out there that it's not super specific. Neon also has PostgREST support.

Rroshposh Do you know if anyone has a working example of querying external MySQL DB from C...

Matt SilverlockOP•11/29/23, 12:26 PM

None of the MySQL drivers work (yet) in Workers. You will need a serverless driver and thus a MySQL provider that allows HTTP/WS access - e.g. https://docs.pingcap.com/tidbcloud/integrate-tidbcloud-with-cloudflare

Integrate TiDB Cloud with Cloudflare

Learn how to deploy Cloudflare Workers with TiDB Cloud.

kian•11/29/23, 12:28 PM

There's https://github.com/nora-soderlund/cloudflare-mysql which is https://github.com/mysqljs/mysql adapted by a community member for the TCP Sockets API

kian•11/29/23, 12:29 PM

It has the same downsides as Postgres w/o Hyperdrive

Kkian There's <https://github.com/nora-soderlund/cloudflare-mysql> which is <https://g...

Matt SilverlockOP•11/29/23, 12:39 PM

@nora have you attempted to upstream the sockets support?

Original message was deleted

Matt SilverlockOP•11/29/23, 12:42 PM

Yes.

Matt SilverlockOP•11/29/23, 1:01 PM

The one we’re looking to contribute to is https://github.com/sidorares/node-mysql2

GitHub

GitHub - sidorares/node-mysql2: :zap: fast mysqljs/mysql compatible...

fast mysqljs/mysql compatible mysql driver for node.js - GitHub - sidorares/node-mysql2:

fast mysqljs/mysql compatible mysql driver for node.js

Original message was deleted

Matt SilverlockOP•11/29/23, 2:00 PM

Same as Postgres libs to get SCRAM working. Does yours require or compat?

James•12/8/23, 12:00 AM

Looks like there's some exciting new developments in the Cloudflare dashboard for Hyperdrive

https://dash.cloudflare.com/?to=/:account/workers/hyperdrive

Paul Shriner•12/8/23, 3:58 AM

I have been spinning wheels on this for a good part of the day. I have HyperDrive (cockroachDB/pg) working with two workers, for whatever reason and I can't figure out what it is, when I run client.connect, it just breaks out of the routine. it doesn't throw an error.

The codes are super duper vanilla:

How do I track this down? It really feels like it has to be config related, but my wrangler files are identical between the three workers.

PPaul Shriner I have been spinning wheels on this for a good part of the day. I have HyperDri...

Matt SilverlockOP•12/8/23, 11:10 AM

Can you share more details?

what version of node-Postgres? Definitely the same with both?
your binding config?
your hyperdrive config ID?

Mitya•12/8/23, 11:43 AM

Hi all. I'm using Hyperdrive with Postgres via

node-postgres

node-postgres

. I read somewhere that one benefit of Hyperdrive is it handles pooling for you. Does this mean that I can forget about generating/managing pools manually in my JS, and just leave that to Hyperdrive? i.e. I can just use

client.query()

client.query()

rather than

pool.query()

pool.query()

, or am I wrong here?

Paul Shriner•12/8/23, 11:57 AM

Hey Matt!

Here is the binding config.. The weird thing is that all of the workers are hitting the exact same DB and instance.

hyperdrive = [
  { binding="REWARDS", id="40161c14328e4af193ee63631756f6ab" }
]

hyperdrive = [
  { binding="REWARDS", id="40161c14328e4af193ee63631756f6ab" }
]

I have attached screen shots of the Hyperdrive config.

Screen_Shot_2023-12-08_at_6.54.40_AM.png

Screen_Shot_2023-12-08_at_6.56.48_AM.png

Paul Shriner•12/8/23, 11:58 AM

Mitya•12/8/23, 12:11 PM

Why does Wrangler tell me to put the PG connection string (for Hyperdrive use) in my

wrangler.toml

wrangler.toml

, given that this file should be committed? This would mean committing sensitive information. Is there a way instead to have the connection string in

.dev.vars

.dev.vars

kian•12/8/23, 12:15 PM

What is the error message that tells you to do that?

Mitya•12/8/23, 12:17 PM

ERROR: In development, you should use a local postgres connection string to emulate hyperdrive functionality. Please setup postgres locally and set the value of "HYPERDRIVE"'s "localConnectionString" to the postgres connection string in your wrangler.toml

Mitya•12/8/23, 12:18 PM

My dev DB lives on Heroku, so it's a remote DB, not locally-hosted

MMitya My dev DB lives on Heroku, so it's a remote DB, not locally-hosted

Max (@rozenmd)•12/8/23, 2:27 PM

can you file a feature request for that? https://github.com/cloudflare/workers-sdk/issues/new?assignees=&labels=enhancement&projects=&template=feature-request.yaml&title=%F0%9F%9A%80+Feature+Request%3A

MMitya Hi all. I'm using Hyperdrive with Postgres via `node-postgres`. I read somewhere...

Matt SilverlockOP•12/8/23, 2:28 PM

You don’t need to manage the pool in the client. Further, because Workers are typically short lived, your “pool” wouldn’t last any longer than your Worker invocation. Hyperdrive’s pooling is much, much more robust.

JJames Looks like there's some exciting new developments in the Cloudflare dashboard fo...

Sam•12/8/23, 3:23 PM

Letsgoooo, finally! Thanks team

MMax (@rozenmd)can you file a feature request for that? https://github.com/cloudflare/workers-s...

Mitya•12/8/23, 5:44 PM

Will do.

MMatt Silverlock You don’t need to manage the pool in the client. Further, because Workers are ty...

Mitya•12/8/23, 5:44 PM

Great! Thanks.

MMax (@rozenmd)can you file a feature request for that? https://github.com/cloudflare/workers-s...

Mitya•12/8/23, 5:47 PM

https://github.com/cloudflare/workers-sdk/issues/4578

Mitya•12/8/23, 5:53 PM

Is it possible to load one TOML in local dev and another for live? I'm aware that the TOML files supports environments, but I'm trying to circumvent this issue. Guess not if the file must be named exactly

wrangler.toml

wrangler.toml

Max (@rozenmd)•12/8/23, 6:08 PM

you can pass a path to the config file

Max (@rozenmd)•12/8/23, 6:09 PM

--config
https://developers.cloudflare.com/workers/wrangler/commands/

Commands - Wrangler · Cloudflare Workers docs

Create, develop, and deploy your Cloudflare Workers with Wrangler commands.