Not yet, no. The ClickHouse protocol doesn’t have a specification: https://clickhouse.com/docs/en/in

Not yet, no. The ClickHouse protocol doesn’t have a specification: https://clickhouse.com/docs/en/interfaces/tcp

Native Interface (TCP) | ClickHouse Docs

The native protocol is used in the command-line client, for inter-server communication during distributed query processing, and also in other C++ programs. Unfortunately, native ClickHouse protocol does not have formal specification yet, but it can be reverse-engineered from ClickHouse source code (starting around here) and/or by intercepting an...

Aarbfay I know more databases are coming later, but may I still ask if ClickHouse is on ...

Matt SilverlockOP•11/21/23, 12:48 PM

Also: are you querying ClickHouse on every request? I can’t imagine that will be fast regardless.

MMatt Silverlock Also: are you querying ClickHouse on every request? I can’t imagine that will be...

DaniFoldi•11/21/23, 12:53 PM

With hyperdrive’s caching that could become possible meowwink

DDaniFoldi With hyperdrive’s caching that could become possible <:meowwink:3804724056778342...

Matt SilverlockOP•11/21/23, 1:20 PM

The caching relies on the protocol being specified (and a stable parser being available, plus a JS driver). But I’m asking as if you’re instead just querying it periodically/in a Cron/etc, the HTTP interface for ClickHouse is likely a more straightforward approach right now.

MMatt Silverlock Not yet, no. The ClickHouse protocol doesn’t have a specification: https://click...

arbfay•11/21/23, 2:26 PM

Ah I didn't know. I guess an experiment with their PostgreSQL protocol is necessary

MMatt Silverlock Also: are you querying ClickHouse on every request? I can’t imagine that will be...

arbfay•11/21/23, 2:28 PM

Sry I'm not sure I see why you think this is a problem.
ClickHouse for OLAP is very performant. I use it for about 600 dashboards (for different users), and yes queries every time (but cached with a worker). It's also what powers Cloudflare's Analytics Engine IIRC.

arbfay•11/21/23, 2:29 PM

It's got an HTTP endpoint to run queries, and honestly I might just stick to the Worker for caching + HTTP. I guess the most interesting part for me was the connection pooling.

Aarbfay Sry I'm not sure I see why you think this is a problem. ClickHouse for OLAP is v...

Matt SilverlockOP•11/21/23, 2:43 PM

I am very aware of how ClickHouse performs. What I am trying to point out is that if you rely on native ClickHouse support you may be waiting a long time, and that CH’s baseline performance may be “fast” but isn’t commonly sub-second for anything but the most simple queries.

Instead, using a HTTP interface will likely get you moving in the meantime, or trying their Postgres interface (untested).

MMatt Silverlock I am very aware of how ClickHouse performs. What I am trying to point out is tha...

arbfay•11/21/23, 3:06 PM

Yeah, I use their HTTP interface since Workers didn't have TCP support yet when we decided to use it and I couldn't be bothered with creating and maintaining a HTTP proxy for any other OLAP/time-series DB. Performance-wise, the queries seem to run fast enough (20-200ms, we really aren't the same size as CF), helped with caching, but the latency is spiky (I suspect its due to making new the connections but haven't investigated yet).

arbfay•11/21/23, 3:06 PM

Anyway, it's all good to know. Might try the Postgres interface some time later

Aarbfay Yeah, I use their HTTP interface since Workers didn't have TCP support yet when ...

Cyb3r-Jak3•11/21/23, 3:28 PM

Workers have outbound TCP connections.

CCyb3r-Jak3 Workers have outbound TCP connections.

arbfay•11/21/23, 3:33 PM

Now, yes. Not last year.

MMatt Silverlock Hyperdrive doesn’t accept arbitrary client connections itself today. It’s design...

DanielZero•11/25/23, 10:56 AM

It's not so supabase specific, actually. One of their core principles is using existing or their own open source software. In this case it's https://github.com/PostgREST/postgrest

DDanielZero It's not so supabase specific, actually. One of their core principles is using e...

Matt SilverlockOP•11/25/23, 10:57 AM

I mention pgrest in my comment.

DanielZero•11/25/23, 10:59 AM

Yeah I know, just wanted to throw out there that it's not super specific. Neon also has PostgREST support.

Rroshposh Do you know if anyone has a working example of querying external MySQL DB from C...

Matt SilverlockOP•11/29/23, 12:26 PM

None of the MySQL drivers work (yet) in Workers. You will need a serverless driver and thus a MySQL provider that allows HTTP/WS access - e.g. https://docs.pingcap.com/tidbcloud/integrate-tidbcloud-with-cloudflare

Integrate TiDB Cloud with Cloudflare

Learn how to deploy Cloudflare Workers with TiDB Cloud.

kian•11/29/23, 12:28 PM

There's https://github.com/nora-soderlund/cloudflare-mysql which is https://github.com/mysqljs/mysql adapted by a community member for the TCP Sockets API

kian•11/29/23, 12:29 PM

It has the same downsides as Postgres w/o Hyperdrive

Kkian There's <https://github.com/nora-soderlund/cloudflare-mysql> which is <https://g...

Matt SilverlockOP•11/29/23, 12:39 PM

@nora have you attempted to upstream the sockets support?

Original message was deleted

Matt SilverlockOP•11/29/23, 12:42 PM

Yes.

Matt SilverlockOP•11/29/23, 1:01 PM

The one we’re looking to contribute to is https://github.com/sidorares/node-mysql2

GitHub

GitHub - sidorares/node-mysql2: :zap: fast mysqljs/mysql compatible...

fast mysqljs/mysql compatible mysql driver for node.js - GitHub - sidorares/node-mysql2:

fast mysqljs/mysql compatible mysql driver for node.js

Original message was deleted

Matt SilverlockOP•11/29/23, 2:00 PM

Same as Postgres libs to get SCRAM working. Does yours require

nodejs

nodejs

node

node

compat?

James•12/8/23, 12:00 AM

Looks like there's some exciting new developments in the Cloudflare dashboard for Hyperdrive

https://dash.cloudflare.com/?to=/:account/workers/hyperdrive

Paul Shriner•12/8/23, 3:58 AM

I have been spinning wheels on this for a good part of the day. I have HyperDrive (cockroachDB/pg) working with two workers, for whatever reason and I can't figure out what it is, when I run client.connect, it just breaks out of the routine. it doesn't throw an error.

The codes are super duper vanilla:

const client = new Client({ connectionString: env.REWARDS.connectionString });

try{
    await client.connect();
} catch (ex){
    console.log(ex)
}

const client = new Client({ connectionString: env.REWARDS.connectionString });

try{
    await client.connect();
} catch (ex){
    console.log(ex)
}

How do I track this down? It really feels like it has to be config related, but my wrangler files are identical between the three workers.

PPaul Shriner I have been spinning wheels on this for a good part of the day. I have HyperDri...

Matt SilverlockOP•12/8/23, 11:10 AM

Can you share more details?

what version of node-Postgres? Definitely the same with both?
your binding config?
your hyperdrive config ID?

Mitya•12/8/23, 11:43 AM

Hi all. I'm using Hyperdrive with Postgres via

node-postgres

node-postgres

. I read somewhere that one benefit of Hyperdrive is it handles pooling for you. Does this mean that I can forget about generating/managing pools manually in my JS, and just leave that to Hyperdrive? i.e. I can just use

client.query()

client.query()

rather than

pool.query()

pool.query()

, or am I wrong here?

Paul Shriner•12/8/23, 11:57 AM

Hey Matt!

Here is the binding config.. The weird thing is that all of the workers are hitting the exact same DB and instance.

hyperdrive = [
  { binding="REWARDS", id="40161c14328e4af193ee63631756f6ab" }
]

hyperdrive = [
  { binding="REWARDS", id="40161c14328e4af193ee63631756f6ab" }
]

I have attached screen shots of the Hyperdrive config.

Screen_Shot_2023-12-08_at_6.54.40_AM.png

Screen_Shot_2023-12-08_at_6.56.48_AM.png

Paul Shriner•12/8/23, 11:58 AM

Mitya•12/8/23, 12:11 PM

Why does Wrangler tell me to put the PG connection string (for Hyperdrive use) in my

wrangler.toml

wrangler.toml

, given that this file should be committed? This would mean committing sensitive information. Is there a way instead to have the connection string in

.dev.vars

.dev.vars

kian•12/8/23, 12:15 PM

What is the error message that tells you to do that?

Mitya•12/8/23, 12:17 PM

ERROR: In development, you should use a local postgres connection string to emulate hyperdrive functionality. Please setup postgres locally and set the value of "HYPERDRIVE"'s "localConnectionString" to the postgres connection string in your wrangler.toml

Mitya•12/8/23, 12:18 PM

My dev DB lives on Heroku, so it's a remote DB, not locally-hosted

MMitya My dev DB lives on Heroku, so it's a remote DB, not locally-hosted

Max (@rozenmd)•12/8/23, 2:27 PM

can you file a feature request for that? https://github.com/cloudflare/workers-sdk/issues/new?assignees=&labels=enhancement&projects=&template=feature-request.yaml&title=%F0%9F%9A%80+Feature+Request%3A

MMitya Hi all. I'm using Hyperdrive with Postgres via `node-postgres`. I read somewhere...

Matt SilverlockOP•12/8/23, 2:28 PM

You don’t need to manage the pool in the client. Further, because Workers are typically short lived, your “pool” wouldn’t last any longer than your Worker invocation. Hyperdrive’s pooling is much, much more robust.

JJames Looks like there's some exciting new developments in the Cloudflare dashboard fo...

Sam•12/8/23, 3:23 PM

Letsgoooo, finally! Thanks team

MMax (@rozenmd)can you file a feature request for that? https://github.com/cloudflare/workers-s...

Mitya•12/8/23, 5:44 PM

Will do.

MMatt Silverlock You don’t need to manage the pool in the client. Further, because Workers are ty...

Mitya•12/8/23, 5:44 PM

Great! Thanks.

MMax (@rozenmd)can you file a feature request for that? https://github.com/cloudflare/workers-s...

Mitya•12/8/23, 5:47 PM

https://github.com/cloudflare/workers-sdk/issues/4578

Mitya•12/8/23, 5:53 PM

Is it possible to load one TOML in local dev and another for live? I'm aware that the TOML files supports environments, but I'm trying to circumvent this issue. Guess not if the file must be named exactly

wrangler.toml

wrangler.toml

Max (@rozenmd)•12/8/23, 6:08 PM

you can pass a path to the config file

Max (@rozenmd)•12/8/23, 6:09 PM

--config
https://developers.cloudflare.com/workers/wrangler/commands/

Commands - Wrangler · Cloudflare Workers docs

Create, develop, and deploy your Cloudflare Workers with Wrangler commands.

Mitya•12/8/23, 6:13 PM

Ah, interesting

Mitya•12/9/23, 12:50 AM

Hi again. My Worker needs to access multiple DBs. Can I set up multiple Hyperdrives under one worker, and how would I model/reference these in the TOML file?

kian•12/9/23, 3:09 AM

Just repeat the [[hyperdrive]] block

Kkian Just repeat the [[hyperdrive]] block

Mitya•12/9/23, 1:48 PM

Right - and then assign a different value to each

binding

binding

param within it, and reference those in my JS to denote which DB to connect to, right?

Mitya•12/9/23, 2:35 PM

When running my worker on CF, for endpoints that talk to Hyperdrive I get an error if I refresh the page soon after getting a response. This seems to happen only for routes that talk to HD, not ones that output a simple, static response. The error is:

A hanging Promise was canceled. This happens when the worker runtime is waiting for a Promise from JavaScript to resolve, but has detected that the Promise cannot possibly ever resolve because all code and events related to the Promise's I/O context have already finished. ✘ [ERROR] Uncaught (in response) Error: The script will never generate a response.

A hanging Promise was canceled. This happens when the worker runtime is waiting for a Promise from JavaScript to resolve, but has detected that the Promise cannot possibly ever resolve because all code and events related to the Promise's I/O context have already finished. ✘ [ERROR] Uncaught (in response) Error: The script will never generate a response.

Any idea what that might be about?

kian•12/9/23, 2:55 PM

Do you have code that reproduces the error?