Cloudflare Developers•3y ago

really excited to play with hyperdrive 🙂 Does it have any usability with rust workers? (ping plz)

really excited to play with hyperdrive

Does it have any usability with rust workers? (ping plz)

DDanTheGoodman really excited to play with hyperdrive 🙂 Does it have any usability with rust w...

Matt Silverlock•9/29/23, 2:23 PM

This should work with a Hyperdrive connection string: https://github.com/cloudflare/workers-rs/tree/main/examples/tokio-postgres

we haven't fully tested, but LMK it does / does not.

kian•9/29/23, 2:35 PM

You'd likely have to use https://docs.rs/worker/latest/worker/struct.Env.html#method.get_binding to access the Hyperdrive binding (for the connection string) as well - but it should be easy enough

Sameer Ali•9/29/23, 8:51 PM

If i have smart placement enabled, does it work with hyperdrive?

Aalexrobinson assuming you mean smart placement, it does not do anything for connections to hy...

Sameer Ali•9/29/23, 9:30 PM

Yeah, smart placement will help minimize the latency in application that makes multiple queries to the database

parisholley•9/30/23, 10:18 PM

can hyperdrive be bound to pages?

James•10/1/23, 2:30 PM

So I have a project that uses Neon for a Postgres DB. My worker currently uses the Neon serverless stuff to connect and works well. I'm also using a bunch of node:buffernode:buffer, node:cryptonode:crypto, etc. stuffs via

nodejs_compat

nodejs_compat

compat flag.

I'd love to use Hyperdrive, but it seems like using

pg

pg

directly requires enabling the older

node_compat

node_compat

, but in doing so, I can no longer use

nodejs_compat

nodejs_compat

and any

node:

node:

stuff in my own code.

Any workaround?

James•10/1/23, 2:40 PM

I guess I could switch to a cutom build and polyfill all the node libs myself... but that's not great DX. Hopefully this can be improved during beta.

ripRigPanic•10/2/23, 4:13 AM

Can hyperdrive be set up for local development to only be used when I run

wrangler dev

wrangler dev

? I want to point it to Postgres running on my machine. I'm looking to continue using RDS HTTP API for everything that is deployed.

JJames So I have a project that uses Neon for a Postgres DB. My worker currently uses t...

Matt Silverlock•10/2/23, 7:07 AM

No workaround yet. Something we’ll fix during the beta.

RripRigPanic Can hyperdrive be set up for local development to only be used when I run `wrang...

Matt Silverlock•10/2/23, 7:07 AM

Not yet, but very soon - we want to support local dev.

Zak @ insertchat.com•10/2/23, 9:47 AM

Hey @Matt, is it possible to use HyperDrive without using workers? My project is already up and running with AdonisJS and I can't move everything to workers atm

ZZak @ insertchat.com Hey @Matt, is it possible to use HyperDrive without using workers? My project is...

Matt Silverlock•10/2/23, 9:48 AM

Not yet - we’re focused on making Workers fast - as Worker’s distributed model + expensive database connection setup weren’t a good combination previously. Where is your app running if not Workers?

Zak @ insertchat.com•10/2/23, 9:49 AM

EC2 server with pm2, a good old nodejs app with AdonisJS as a framework and an Aurora database

kian•10/2/23, 9:51 AM

Will Hyperdrive (and I guess the TCP API in general) be affected by Aegis' dedicated egress IP, or is that use-case going to be covered by the Tunnel integration when that lands?

ZZak @ insertchat.com EC2 server with pm2, a good old nodejs app with AdonisJS as a framework and an A...

Matt Silverlock•10/2/23, 10:19 AM

Interesting: candidly, I would be surprised if Hyperdrive is faster here. Long running EC2 VM and Aurora - Hyperdrive’s caching will be nice in some cases but for AWS-to-AWS you’re going to pay egress + latency to get to us. Would love to say we are a win there but if this is same region…

Kkian Will Hyperdrive (and I guess the TCP API in general) be affected by Aegis' dedic...

Matt Silverlock•10/2/23, 10:20 AM

This - it’s all dependent on plugging into Magic WAN/Tunnel infrastructure (same thing).

Zak @ insertchat.com•10/2/23, 10:26 AM

@Matt Thanks for the answer, the thing is I want to switch to Vectorize at some point, will there be a point when it's open to non workers users? Or do I need to find a way to push data to Vectorize using my node app, then call a CF Worker to get my vectors and process them on my node app?

ZZak @ insertchat.com @Matt Thanks for the answer, the thing is I want to switch to Vectorize at some ...

Matt Silverlock•10/2/23, 10:29 AM

Vectorize and Hyperdrive are entirely separate things. You can use Vectorize through a Worker to expose your own API, or use Vectorize’ own API.

ZZak @ insertchat.com @Matt Thanks for the answer, the thing is I want to switch to Vectorize at some ...

Matt Silverlock•10/2/23, 10:30 AM

https://developers.cloudflare.com/api/operations/vectorize-list-vectorize-indexes

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Zak @ insertchat.com•10/2/23, 10:30 AM

@Matt I understand, I was just hoping for a way to get the full benefits of CF workers without having to rewrite all the code. I guess I will add moving to Workers to the roadmap x)

Zak @ insertchat.com•10/2/23, 10:31 AM

By the way, Vectorize is built on top of what? I can't find the info or any benchmark

Matt Silverlock•10/2/23, 10:32 AM

I would (genuinely) LOVE to understand how you came to understand that you need both - you can’t really benefit from Workers if your app is still directly in EC2. Not sure what architecture you’re thinking here.

Moving your “middleware” (all your database interop code) to Workers can be a good first step though.

ZZak @ insertchat.com By the way, Vectorize is built on top of what? I can't find the info or any benc...

Matt Silverlock•10/2/23, 10:32 AM

Rust, Workers and Durable Objects.

Zak @ insertchat.com•10/2/23, 10:33 AM

Is there any speed benchmark on the way? Already using pgvector right now

ZZak @ insertchat.com Is there any speed benchmark on the way? Already using pgvector right now

Matt Silverlock•10/2/23, 10:35 AM

Not yet. We just launched a beta: things will change too much over the coming weeks & months.

MMatt Silverlock This should work with a Hyperdrive connection string: https://github.com/cloudfl...

kian•10/2/23, 11:36 AM

Good news: this works

Bad news: tokio-postgrestokio-postgres and connection pooling don't like each other, so you have to resort to query_rawquery_raw and whatnot which isn't the nicest.

kian•10/2/23, 11:39 AM

Good news: https://github.com/sfackler/rust-postgres/pull/1067 looks like it might fix that issue

Kkian Good news: this works Bad news: `tokio-postgres` and connection pooling don't l...

Matt Silverlock•10/2/23, 11:50 AM

Thanks for checking!

kian•10/2/23, 11:50 AM

Using that fork, it seems fine - querying my Neon DB is in the realm of 120-140ms

Kkian Good news: https://github.com/sfackler/rust-postgres/pull/1067 looks like it mig...

Matt Silverlock•10/2/23, 11:53 AM

It should. Named prepared statements don’t work with any form of connection pooling as they are connection-scoped in Postgres: https://developers.cloudflare.com/hyperdrive/learning/connect-to-postgres/#transaction-and-statement-support

Connect to PostgreSQL · Hyperdrive

Hyperdrive supports both PostgreSQL and PostgreSQL-compatible databases, as well as popular drivers and the many Object Relational Mapper (ORM) …

kian•10/2/23, 11:53 AM

The accompanying wasm_bindgenwasm_bindgen is at https://gist.github.com/KianNH/66c513ac8c37c3ab8e97d36665394094

kian•10/2/23, 11:55 AM

I'll write up the rest in the Gist later

MMatt Silverlock It should. Named prepared statements don’t work with any form of connection pool...

kian•10/2/23, 11:56 AM

I originally thought I was going mad after using a non-pooled connection string from Neon in Hyperdrive before I realised that's one of the main points of Hyperdrive, connection pooling. meowpuffydizzy

kian•10/2/23, 2:55 PM

It'd be nice if Wrangler made a warning/error when you use local development w/ Hyperdrive bindings. You do get a warning that they're experimental, and that callout is made in the docs, but it's an easy issue to hit with local being the default.

kian•10/2/23, 2:55 PM

Something similar to Queues, which tells you that they're not available in remote mode.

kian•10/2/23, 3:40 PM

Crossposting from #what-i-built

https://kian.org.uk/using-cloudflare-hyperdrive-with-tokio-postgres-in-workers/

Kian's Blog

Using Cloudflare Hyperdrive with tokio-postgres in Workers

Cloudflare recently announced their query caching & connection pooling solution for databases in Workers, Hyperdrive.

Whilst there's plenty of examples of using Hyperdrive with libraries like pg in the documentation, someone in the Cloudflare Developers Discord asked about usability with Rust Workers.

There are examples for using tokio-postgre...

kian•10/3/23, 3:57 PM

I think requiring an explicit port number in the connection string is a bit odd and non-standard, though. Could you default it to 5432 like everyone else?

There's an open PR which will do that when merged.

kian•10/3/23, 4:05 PM

My suspicion is that the id can only be used from within my own account, and so it isn't?

Yep.

Unsmart•10/3/23, 4:06 PM

All of the things you have to enter into

wrangler.toml

wrangler.toml

for bindings will not be secret, I think it would be an extremely bad look if an Internet security company made you enter in a secret value into a config that should be pushed to git