I have a question about how hyperdrive works that I couldn't find an answer to in the blog or docs:

I have a question about how hyperdrive works that I couldn't find an answer to in the blog or docs:

Let's say a customer is in Singapore, the CF worker is Singapore, and the database is in US-East1.

Is there any difference in the network path (and subsequent latency) for our Singapore CF worker using hyperdrive vs using something like the Neon serverless driver? Is it using CF peering connections or something else that would make the latency lower when using CF?

Aandyhats I have a question about how hyperdrive works that I couldn't find an answer to i...

Colin•1/1/24, 7:42 PM

As far as I understand (I don't work for CF or anything), Hyperdrive will do the PG handshake at the edge right near your worker (with very low latency), and Neon serverless will do the PG handshake at the origin. The PG handshake is like 7 (this is a guess) round trips, so the connection establish time is significantly lower

CColin As far as I understand (I don't work for CF or anything), Hyperdrive will do the...

andyhatsOP•1/1/24, 7:53 PM

thanks Colin - yeah that makes sense. I am just wondering if there is also any material difference in the path that the bits take as they route around the globe.

Honestly no idea if it would even make a difference, but it would make sense to me if this route:

USER -> [CF SGP] --CF-PVT-NETWORK--> [CF USA] -> DB US-EAST

USER -> [CF SGP] --CF-PVT-NETWORK--> [CF USA] -> DB US-EAST

was faster than...

USER -> [CF SGP] --Public Internet--> DB US-EAST

USER -> [CF SGP] --Public Internet--> DB US-EAST

Colin•1/1/24, 7:54 PM

As far as I'm aware the packets do go over the public Internet right now, not CF fiber

CColin Account ID: `cfd355bcff7e6af6838743ae944184b0` Hyperdrive ID: `0bbf89fd8f5544b4a...

Colin•1/1/24, 7:55 PM

Wanted to add onto this. We created two Hyperdrives, one with cache, one without. The one with cache consistently has this issue, the one without cache never does. Happy to share exact queries + dev DB to reproduce if someone from CF is interested in taking a look

CColin Wanted to add onto this. We created two Hyperdrives, one with cache, one without...

AJR•1/2/24, 1:57 AM

If you have a schema and queries that reproduce this that you're comfortable sharing, definitely please do!

AAngusMa The hyperdrive page shows nothing in Traditional Chinese, but it works in Englis...

Emilio•1/3/24, 10:14 PM

Looks like the internationalization for zh-TW is not yet complete for the Hyperdrive page. Apologies, and I will try to see if we can have a smoother experience for when this happens.

moody 💭•1/8/24, 4:15 AM

is it possible to get wire compression with hyperdrive to postgres?

moody 💭•1/8/24, 4:15 AM

my data egress costs to hyperdrive are significant on amazon

Mmoody 💭is it possible to get wire compression with hyperdrive to postgres?

Matt Silverlock•1/8/24, 11:39 AM

Does PG have wire compression these days? It was historically disabled (at the SSL level) back in ~2016 due to vulns, and I haven’t seen any protocol level support for compressing result sets land?

MMatt Silverlock Does PG have wire compression these days? It was historically disabled (at the S...

moody 💭•1/8/24, 12:28 PM

yeah after i sent that i found the same thing

moody 💭•1/8/24, 12:28 PM

it's a shame because my egress costs more than compute

moody 💭•1/8/24, 12:30 PM

it seems like i need to put the compute next to the db in aws and hit the compute endpoint instead of a db

Mmoody 💭it seems like i need to put the compute next to the db in aws and hit the comput...

Matt Silverlock•1/8/24, 1:27 PM

You’re still going to incur egress (same fees/rates) either way - that’s the downside of AWS.

moody 💭•1/8/24, 1:28 PM

yeah but i'm fetching a lot of data, computing some small piece of information from it in the worker, and then returning that small piece of information to the client

moody 💭•1/8/24, 1:28 PM

i cache as much as possible in worker memory

Mitya•1/8/24, 5:47 PM

I'm experiencing some really weird behaviour whereby one particular DB query sometimes comes back with rows and sometimes comes back with this:

{
  command: null,
  rowCount: null,
  oid: null,
  rows: Array(0),
  fields: Array(0)
  ...
}

{
  command: null,
  rowCount: null,
  oid: null,
  rows: Array(0),
  fields: Array(0)
  ...
}

This does not happen in another (non-CF) app connecting to the same DB and running the same query. Wrangler/Hyperdrive is the only difference. Does this ring any bells? Are there any query caching issues or behaviours I can tweak? Right now I'm refreshing an endpoint (with no code changes) and sometimes I get data back from the DB and sometimes I get the above.

Any thoughts?

Mitya•1/8/24, 5:49 PM

The above is happening when testing locally, with

--remote

--remote

enabled. I believe this bypasses Hyperdrive entirely, so if the issue relates to CF at all, perhaps it's a Wrangler thing not HD. Either way it's a blocker for me right now.

Mitya•1/8/24, 5:52 PM

Aha interesting - glad to know it's not just me, though sorry it's affecvting you too. How did you disable caching? Presumably this defeats the point of HD in general since it's essentially a DB cacher?

Mitya•1/8/24, 5:58 PM

If I'm right that running a Worker app locally in dev mode bypasses HD entirely anyway (i.e. simply connects to the DB conventionally), then possibly (at least in my case) it's not a HD issue but a Workers one.

Mitya•1/8/24, 6:03 PM

Hmm, info but no option to change settings. Perhaps this has to be done via Wrangler CLI

Mitya•1/8/24, 6:05 PM

And you're right re:

--remote

--remote

implying usage of HD, just checked.

MMitya Hmm, info but no option to change settings. Perhaps this has to be done via Wran...

Mitya•1/8/24, 6:08 PM

Here's the commands to do that

Mitya•1/8/24, 6:10 PM

Anyway I've formally DB sometimes returns rows, sometimes returns nothing up this issue in #workers-help. Gonna disappear for dinner but will check back later/tomorrow in case there's any update on this problem. Hopefully a CF dev can shed some light/a fix.

MMitya [Here's the commands](https://developers.cloudflare.com/workers/wrangler/command...

Colin•1/8/24, 7:37 PM

Hey, just wanted to clarify on this (and Connor is one of my coworkers, thank you connor!!) - I've been talking with xortive in DMs + helping reproduce this. We're currently working around this w/ 2 Hyperdrives (one with caching, one without).

Also confirmed that using SQL queries w/o any parameters (IE doing string concatenation instead of using $1/$2/etc) doesn't trigger the issue; it's only with the parameterized queries

Colin•1/8/24, 7:37 PM

Obviously if string concatenation works for you or not is based on what kind of queries you have (SQL injection and all)

Colin•1/8/24, 7:37 PM

Also got linked this branch of Wrangler: https://github.com/xortive/workers-sdk/tree/malonso/caching-options which provides options to change caching settings . Not sure if includes enabling/disabling as i haven't looked yet, but we're running with two hyperdrives anyways (we have enough cacheable volume that we'd blow up our DB if we disabled it entirely), so I just made a new one with it disabled

MMitya Aha interesting - glad to know it's not just me, though sorry it's affecvting yo...

Matt Silverlock•1/8/24, 11:57 PM

Separate from the caching bug, the connection pooling part of Hyperdrive is a very substantial benefit — otherwise you pay connection setup overhead/latency back to your DB over and over.

Mitya•1/9/24, 10:59 AM

Thanks, everyone! Appreciate the suggestions, @Colin and interesting that it's only parameterised queries. And great news about the fix, @Matt - look forward to getting confirmation that that's live

Mitya•1/9/24, 2:34 PM

This page in the HD docs says you can disable caching "using the wrangler CLI or the Cloudflare dashboard". It then gives this example:

npx wrangler hyperdrive update my-hyperdrive --disable-caching

npx wrangler hyperdrive update my-hyperdrive --disable-caching

However the

--disable-caching

--disable-caching

option isn't documented in the

hyperdrive update

hyperdrive update

docs, and the HD dashboard doesn't seem to provide any means to edit settings (see my hyperdrive). So is this just a case of the docs being out of sync?

Mitya•1/9/24, 4:09 PM

(The docs also don't mention that you need to pass

--origin-password

--origin-password

with the above command, too.)

kian•1/9/24, 4:11 PM

Yeah, the Wrangler docs tend to get outdated quickly

Mitya•1/9/24, 4:11 PM

Is that why I'm getting "Error - Unknown arguments: disable-caching, disableCaching" i.e. this option isn't live yet?

Mitya•1/9/24, 4:16 PM

Yeah that command given by the docs is definitely not working, even with latest Wrangler. Command I'm using is

wrangler hyperdrive update my-hyperdrive-name --origin-password mypassword --disable-caching

wrangler hyperdrive update my-hyperdrive-name --origin-password mypassword --disable-caching

kian•1/9/24, 4:19 PM

it’s —caching-disabled (pass true/false)

kian•1/9/24, 4:19 PM

docs are wrong

Mitya•1/9/24, 4:22 PM

I took that to be a different option, which tells you whether caching is disabld but doesn't allow you to set it. Are you saying that I can set it via that option, and if so do I just follow it with true/false?

Mitya•1/9/24, 4:22 PM

Ah I see your edit.

kian•1/9/24, 4:22 PM

honestly I'd recommend doing over using the docs

kian•1/9/24, 4:23 PM

the docs fall out of sync

Mitya•1/9/24, 4:23 PM

Ah OK cool. Wherever I found the option I didn't see the boolean reference following it (or it wasn't there). I see it is there in

--help

--help

Mitya•1/9/24, 4:31 PM

I've edited the docs and submitted a PR.

MMatt Silverlock We’re working with Mongo. Their driver is not simple to run in a (proper) server...

bryan•1/10/24, 4:40 AM

Is there any timeline for this support? Or even any other workarounds. Their data API is definitely not super performant (random 20sec spikes) and it doesn't support some crucial basics...like transactions (!)

Bbryan Is there any timeline for this support? Or even any other workarounds. Their dat...

Matt Silverlock•1/10/24, 10:16 AM

No. It’s dependent on Mongo’s driver — which we don’t control.

bryan•1/10/24, 2:15 PM

understood, but was hopeful you'd just enable TCP in a place I could write my own

any other workarounds you are aware of? do you know anyone successfully using an external connection pooler like prisma accelerate? i came across that last night

Bbryan understood, but was hopeful you'd just enable TCP in a place I could write my ow...

Matt Silverlock•1/10/24, 2:32 PM

You can write your own. Hyperdrive uses the socket API in Workers - the Postgres (and soon) MySQL drivers we support adopted that. MongoDB’s driver a) doesn’t yet run in any edge serverless runtime and b) holds a lot of client state. Both of these are big blockers to us.

https://blog.cloudflare.com/hyperdrive-making-regional-databases-feel-distributed

The Cloudflare Blog

Hyperdrive: making databases feel like they’re global

Hyperdrive makes accessing your existing databases from Cloudflare Workers, wherever they are running, hyper fast.

Bbryan understood, but was hopeful you'd just enable TCP in a place I could write my ow...

Matt Silverlock•1/10/24, 2:33 PM

IIRC Prisma Accelerate also doesn’t support Mongo, likely for many of the same architectural reasons (running a highly stateful Mongo proxy is non-trivial)