is it possible to get wire compression with hyperdrive to postgres?
is it possible to get wire compression with hyperdrive to postgres?
Mmy data egress costs to hyperdrive are significant on amazon
MDoes PG have wire compression these days? It was historically disabled (at the SSL level) back in ~2016 due to vulns, and I haven’t seen any protocol level support for compressing result sets land?
Myeah after i sent that i found the same thing
Mit's a shame because my egress costs more than compute
Mit seems like i need to put the compute next to the db in aws and hit the compute endpoint instead of a db
MYou’re still going to incur egress (same fees/rates) either way - that’s the downside of AWS.
Myeah but i'm fetching a lot of data, computing some small piece of information from it in the worker, and then returning that small piece of information to the client
Mi cache as much as possible in worker memory
MI'm experiencing some really weird behaviour whereby one particular DB query sometimes comes back with rows and sometimes comes back with this:
This does not happen in another (non-CF) app connecting to the same DB and running the same query. Wrangler/Hyperdrive is the only difference. Does this ring any bells? Are there any query caching issues or behaviours I can tweak? Right now I'm refreshing an endpoint (with no code changes) and sometimes I get data back from the DB and sometimes I get the above.
Any thoughts?
This does not happen in another (non-CF) app connecting to the same DB and running the same query. Wrangler/Hyperdrive is the only difference. Does this ring any bells? Are there any query caching issues or behaviours I can tweak? Right now I'm refreshing an endpoint (with no code changes) and sometimes I get data back from the DB and sometimes I get the above.
Any thoughts?
MThe above is happening when testing locally, with
--remote enabled. I believe this bypasses Hyperdrive entirely, so if the issue relates to CF at all, perhaps it's a Wrangler thing not HD. Either way it's a blocker for me right now.MAha interesting - glad to know it's not just me, though sorry it's affecvting you too. How did you disable caching? Presumably this defeats the point of HD in general since it's essentially a DB cacher?
MIf I'm right that running a Worker app locally in dev mode bypasses HD entirely anyway (i.e. simply connects to the DB conventionally), then possibly (at least in my case) it's not a HD issue but a Workers one.
MHmm, info but no option to change settings. Perhaps this has to be done via Wrangler CLI
MAnd you're right re:
--remote implying usage of HD, just checked.
MHere's the commands to do that
MAnyway I've formally DB sometimes returns rows, sometimes returns nothing up this issue in #workers-help. Gonna disappear for dinner but will check back later/tomorrow in case there's any update on this problem. Hopefully a CF dev can shed some light/a fix.
CHey, just wanted to clarify on this (and Connor is one of my coworkers, thank you connor!!) - I've been talking with xortive in DMs + helping reproduce this. We're currently working around this w/ 2 Hyperdrives (one with caching, one without).
Also confirmed that using SQL queries w/o any parameters (IE doing string concatenation instead of using $1/$2/etc) doesn't trigger the issue; it's only with the parameterized queries
Also confirmed that using SQL queries w/o any parameters (IE doing string concatenation instead of using $1/$2/etc) doesn't trigger the issue; it's only with the parameterized queries
CObviously if string concatenation works for you or not is based on what kind of queries you have (SQL injection and all)
CAlso got linked this branch of Wrangler: https://github.com/xortive/workers-sdk/tree/malonso/caching-options which provides options to change caching settings . Not sure if includes enabling/disabling as i haven't looked yet, but we're running with two hyperdrives anyways (we have enough cacheable volume that we'd blow up our DB if we disabled it entirely), so I just made a new one with it disabled
MSeparate from the caching bug, the connection pooling part of Hyperdrive is a very substantial benefit — otherwise you pay connection setup overhead/latency back to your DB over and over.
MThanks, everyone! Appreciate the suggestions, @Colin and interesting that it's only parameterised queries. And great news about the fix, @Matt - look forward to getting confirmation that that's live 
MThis page in the HD docs says you can disable caching "using the wrangler CLI or the Cloudflare dashboard". It then gives this example:
However the
However the
--disable-caching option isn't documented in the hyperdrive update docs, and the HD dashboard doesn't seem to provide any means to edit settings (see my hyperdrive). So is this just a case of the docs being out of sync?
KYeah, the Wrangler docs tend to get outdated quickly
MIs that why I'm getting "Error - Unknown arguments: disable-caching, disableCaching" i.e. this option isn't live yet?
MYeah that command given by the docs is definitely not working, even with latest Wrangler. Command I'm using is
wrangler hyperdrive update my-hyperdrive-name --origin-password mypassword --disable-cachingKit’s —caching-disabled (pass true/false)
Kdocs are wrong
MI took that to be a different option, which tells you whether caching is disabld but doesn't allow you to set it. Are you saying that I can set it via that option, and if so do I just follow it with true/false?
MAh I see your edit.
KKhonestly I'd recommend doing
wrangler <cmd> --help over using the docsKthe docs fall out of sync
MAh OK cool. Wherever I found the
--caching-disabled option I didn't see the boolean reference following it (or it wasn't there). I see it is there in --helpI've edited the docs and submitted a PR.
BIs there any timeline for this support? Or even any other workarounds. Their data API is definitely not super performant (random 20sec spikes) and it doesn't support some crucial basics...like transactions (!)
MNo. It’s dependent on Mongo’s driver — which we don’t control.
Bunderstood, but was hopeful you'd just enable TCP in a place I could write my own 
any other workarounds you are aware of? do you know anyone successfully using an external connection pooler like prisma accelerate? i came across that last night
any other workarounds you are aware of? do you know anyone successfully using an external connection pooler like prisma accelerate? i came across that last nightMYou can write your own. Hyperdrive uses the socket API in Workers - the Postgres (and soon) MySQL drivers we support adopted that. MongoDB’s driver a) doesn’t yet run in any edge serverless runtime and b) holds a lot of client state. Both of these are big blockers to us.
https://blog.cloudflare.com/hyperdrive-making-regional-databases-feel-distributed
https://blog.cloudflare.com/hyperdrive-making-regional-databases-feel-distributed
The Cloudflare Blog
Hyperdrive makes accessing your existing databases from Cloudflare Workers, wherever they are running, hyper fast.
MIIRC Prisma Accelerate also doesn’t support Mongo, likely for many of the same architectural reasons (running a highly stateful Mongo proxy is non-trivial)
BGood to know re: socket API, am I able to create connections that survive worker requests and selectively pool? Not seeing anything in the API docs for that specifically
BI'll stop bugging, but are you aware of any active mongodb projects (from mong) to make the driver edge-compatible? Happy to go bark up their tree
M
Is there an issue with HD right now? Suddenly all my requests are giving this
MTried restarting the server, same deal
MIt's just come back up (with no code changes) but for ~5-10 mins that was all I was getting for every DB request.
