👋 we're seeing what we believe to be inconsistent caching in Hyperdrive when repeatedly sending a

we're seeing what we believe to be inconsistent caching in Hyperdrive

when repeatedly sending a relatively simple SQL query (looks just like

SELECT data FROM my_table WHERE primary_key = 1234;

SELECT data FROM my_table WHERE primary_key = 1234;

) with a large result (

data

data

is a couple hundred KB in size), Hyperdrive sometimes responds correctly (with one row) and sometimes responds incorrectly (with no rows)

It also seems to go in runs - we'll get correct responses for a minute, then incorrect responses for a minute, and so on. I've only ever been able to reproduce this when going through Hyperdrive (with a minute TTL cache), never when connecting directly to the DB

Unfortunately we rely pretty heavily on Hyperdrive caching to not overload the origin DB so disabling cache isn't an option here

MMatt Silverlock Not in the near term, but we're talking with MongoDB. Their driver architecture ...

Gagan Suie•12/30/23, 3:51 AM

At the moment there's a mTLS to TLS issue right? Kind of tired of using mongoDB data api. Any plans to fix the driver issue for workers?

GGagan Suie At the moment there's a mTLS to TLS issue right? Kind of tired of using mongoDB ...

Matt Silverlock•12/30/23, 4:16 AM

We’re working with Mongo. Their driver is not simple to run in a (proper) serverless environment

CColin 👋 we're seeing what we believe to be inconsistent caching in Hyperdrive when r...

Matt Silverlock•12/30/23, 4:16 AM

What’s your account ID & hyperdrive config ID?

MMatt Silverlock What’s your account ID & hyperdrive config ID?

ColinOP•12/30/23, 4:18 AM

Account ID:

cfd355bcff7e6af6838743ae944184b0

cfd355bcff7e6af6838743ae944184b0

Hyperdrive ID:

0bbf89fd8f5544b4a313f26006174018

0bbf89fd8f5544b4a313f26006174018

andyhats•12/31/23, 6:58 PM

I have a question about how hyperdrive works that I couldn't find an answer to in the blog or docs:

Let's say a customer is in Singapore, the CF worker is Singapore, and the database is in US-East1.

Is there any difference in the network path (and subsequent latency) for our Singapore CF worker using hyperdrive vs using something like the Neon serverless driver? Is it using CF peering connections or something else that would make the latency lower when using CF?

Aandyhats I have a question about how hyperdrive works that I couldn't find an answer to i...

ColinOP•1/1/24, 7:42 PM

As far as I understand (I don't work for CF or anything), Hyperdrive will do the PG handshake at the edge right near your worker (with very low latency), and Neon serverless will do the PG handshake at the origin. The PG handshake is like 7 (this is a guess) round trips, so the connection establish time is significantly lower

CColin As far as I understand (I don't work for CF or anything), Hyperdrive will do the...

andyhats•1/1/24, 7:53 PM

thanks Colin - yeah that makes sense. I am just wondering if there is also any material difference in the path that the bits take as they route around the globe.

Honestly no idea if it would even make a difference, but it would make sense to me if this route:

USER -> [CF SGP] --CF-PVT-NETWORK--> [CF USA] -> DB US-EAST

USER -> [CF SGP] --CF-PVT-NETWORK--> [CF USA] -> DB US-EAST

was faster than...

USER -> [CF SGP] --Public Internet--> DB US-EAST

USER -> [CF SGP] --Public Internet--> DB US-EAST

ColinOP•1/1/24, 7:54 PM

As far as I'm aware the packets do go over the public Internet right now, not CF fiber

CColin Account ID: `cfd355bcff7e6af6838743ae944184b0` Hyperdrive ID: `0bbf89fd8f5544b4a...

ColinOP•1/1/24, 7:55 PM

Wanted to add onto this. We created two Hyperdrives, one with cache, one without. The one with cache consistently has this issue, the one without cache never does. Happy to share exact queries + dev DB to reproduce if someone from CF is interested in taking a look

CColin Wanted to add onto this. We created two Hyperdrives, one with cache, one without...

AJR•1/2/24, 1:57 AM

If you have a schema and queries that reproduce this that you're comfortable sharing, definitely please do!

AAngusMa The hyperdrive page shows nothing in Traditional Chinese, but it works in Englis...

Emilio•1/3/24, 10:14 PM

Looks like the internationalization for zh-TW is not yet complete for the Hyperdrive page. Apologies, and I will try to see if we can have a smoother experience for when this happens.

moody 💭•1/8/24, 4:15 AM

is it possible to get wire compression with hyperdrive to postgres?

moody 💭•1/8/24, 4:15 AM

my data egress costs to hyperdrive are significant on amazon

Mmoody 💭is it possible to get wire compression with hyperdrive to postgres?

Matt Silverlock•1/8/24, 11:39 AM

Does PG have wire compression these days? It was historically disabled (at the SSL level) back in ~2016 due to vulns, and I haven’t seen any protocol level support for compressing result sets land?

MMatt Silverlock Does PG have wire compression these days? It was historically disabled (at the S...

moody 💭•1/8/24, 12:28 PM

yeah after i sent that i found the same thing

moody 💭•1/8/24, 12:28 PM

it's a shame because my egress costs more than compute

moody 💭•1/8/24, 12:30 PM

it seems like i need to put the compute next to the db in aws and hit the compute endpoint instead of a db

Mmoody 💭it seems like i need to put the compute next to the db in aws and hit the comput...

Matt Silverlock•1/8/24, 1:27 PM

You’re still going to incur egress (same fees/rates) either way - that’s the downside of AWS.

moody 💭•1/8/24, 1:28 PM

yeah but i'm fetching a lot of data, computing some small piece of information from it in the worker, and then returning that small piece of information to the client

moody 💭•1/8/24, 1:28 PM

i cache as much as possible in worker memory

Mitya•1/8/24, 5:47 PM

I'm experiencing some really weird behaviour whereby one particular DB query sometimes comes back with rows and sometimes comes back with this:

{
  command: null,
  rowCount: null,
  oid: null,
  rows: Array(0),
  fields: Array(0)
  ...
}

{
  command: null,
  rowCount: null,
  oid: null,
  rows: Array(0),
  fields: Array(0)
  ...
}

This does not happen in another (non-CF) app connecting to the same DB and running the same query. Wrangler/Hyperdrive is the only difference. Does this ring any bells? Are there any query caching issues or behaviours I can tweak? Right now I'm refreshing an endpoint (with no code changes) and sometimes I get data back from the DB and sometimes I get the above.

Any thoughts?

Mitya•1/8/24, 5:49 PM

The above is happening when testing locally, with

--remote

--remote

enabled. I believe this bypasses Hyperdrive entirely, so if the issue relates to CF at all, perhaps it's a Wrangler thing not HD. Either way it's a blocker for me right now.

Mitya•1/8/24, 5:52 PM

Aha interesting - glad to know it's not just me, though sorry it's affecvting you too. How did you disable caching? Presumably this defeats the point of HD in general since it's essentially a DB cacher?

Mitya•1/8/24, 5:58 PM

If I'm right that running a Worker app locally in dev mode bypasses HD entirely anyway (i.e. simply connects to the DB conventionally), then possibly (at least in my case) it's not a HD issue but a Workers one.

Mitya•1/8/24, 6:03 PM

Hmm, info but no option to change settings. Perhaps this has to be done via Wrangler CLI

Mitya•1/8/24, 6:05 PM

And you're right re:

--remote

--remote

implying usage of HD, just checked.

MMitya Hmm, info but no option to change settings. Perhaps this has to be done via Wran...

Mitya•1/8/24, 6:08 PM

Here's the commands to do that

Mitya•1/8/24, 6:10 PM

Anyway I've formally DB sometimes returns rows, sometimes returns nothing up this issue in #workers-help. Gonna disappear for dinner but will check back later/tomorrow in case there's any update on this problem. Hopefully a CF dev can shed some light/a fix.

MMitya [Here's the commands](https://developers.cloudflare.com/workers/wrangler/command...

ColinOP•1/8/24, 7:37 PM

Hey, just wanted to clarify on this (and Connor is one of my coworkers, thank you connor!!) - I've been talking with xortive in DMs + helping reproduce this. We're currently working around this w/ 2 Hyperdrives (one with caching, one without).

Also confirmed that using SQL queries w/o any parameters (IE doing string concatenation instead of using $1/$2/etc) doesn't trigger the issue; it's only with the parameterized queries

ColinOP•1/8/24, 7:37 PM

Obviously if string concatenation works for you or not is based on what kind of queries you have (SQL injection and all)

ColinOP•1/8/24, 7:37 PM

Also got linked this branch of Wrangler: https://github.com/xortive/workers-sdk/tree/malonso/caching-options which provides options to change caching settings . Not sure if includes enabling/disabling as i haven't looked yet, but we're running with two hyperdrives anyways (we have enough cacheable volume that we'd blow up our DB if we disabled it entirely), so I just made a new one with it disabled

MMitya Aha interesting - glad to know it's not just me, though sorry it's affecvting yo...

Matt Silverlock•1/8/24, 11:57 PM

Separate from the caching bug, the connection pooling part of Hyperdrive is a very substantial benefit — otherwise you pay connection setup overhead/latency back to your DB over and over.

Mitya•1/9/24, 10:59 AM

Thanks, everyone! Appreciate the suggestions, @Colin and interesting that it's only parameterised queries. And great news about the fix, @Matt - look forward to getting confirmation that that's live

Mitya•1/9/24, 2:34 PM

This page in the HD docs says you can disable caching "using the wrangler CLI or the Cloudflare dashboard". It then gives this example:

npx wrangler hyperdrive update my-hyperdrive --disable-caching

npx wrangler hyperdrive update my-hyperdrive --disable-caching

However the

--disable-caching

--disable-caching

option isn't documented in the

hyperdrive update

hyperdrive update

docs, and the HD dashboard doesn't seem to provide any means to edit settings (see my hyperdrive). So is this just a case of the docs being out of sync?

Mitya•1/9/24, 4:09 PM

(The docs also don't mention that you need to pass

--origin-password

--origin-password

with the above command, too.)

kian•1/9/24, 4:11 PM

Yeah, the Wrangler docs tend to get outdated quickly

Mitya•1/9/24, 4:11 PM

Is that why I'm getting "Error - Unknown arguments: disable-caching, disableCaching" i.e. this option isn't live yet?

Mitya•1/9/24, 4:16 PM

Yeah that command given by the docs is definitely not working, even with latest Wrangler. Command I'm using is

wrangler hyperdrive update my-hyperdrive-name --origin-password mypassword --disable-caching

wrangler hyperdrive update my-hyperdrive-name --origin-password mypassword --disable-caching

kian•1/9/24, 4:19 PM

it’s —caching-disabled (pass true/false)

kian•1/9/24, 4:19 PM

docs are wrong

Mitya•1/9/24, 4:22 PM

I took that to be a different option, which tells you whether caching is disabld but doesn't allow you to set it. Are you saying that I can set it via that option, and if so do I just follow it with true/false?

Mitya•1/9/24, 4:22 PM

Ah I see your edit.

kian•1/9/24, 4:22 PM

--caching-disabled  Whether caching query results is disabled for this Hyperdrive config  [boolean]

--caching-disabled  Whether caching query results is disabled for this Hyperdrive config  [boolean]

kian•1/9/24, 4:22 PM

honestly I'd recommend doing

wrangler <cmd> --help

wrangler <cmd> --help

over using the docs

kian•1/9/24, 4:23 PM

the docs fall out of sync

Mitya•1/9/24, 4:23 PM

Ah OK cool. Wherever I found the

--caching-disabled

--caching-disabled

option I didn't see the boolean reference following it (or it wasn't there). I see it is there in

--help

--help

Mitya•1/9/24, 4:31 PM

I've edited the docs and submitted a PR.