Cloudflare Developers•3y ago

Cool Is time travel built on ZFS?

Cool
Is time travel built on ZFS?

IIsaac McFadyen I'm unsure, and I doubt anyone will be able to tell you since nothing is concret...

Jacob Wright•9/28/23, 6:14 PM

Good point.

JJacob Wright Will we need to do the same at general release?

Matt Silverlock•9/28/23, 7:13 PM

Not if you are on the beta backend.

Aabdellah Cool Is time travel built on ZFS?

Matt Silverlock•9/28/23, 7:14 PM

No. ZFS snapshots don’t work like that (and way too low level). D1 has its own storage subsystem that allows this to work.

MMatt Silverlock At "100k databases" there are other things that make you want to use WfP in term...

nclevenger•9/28/23, 7:27 PM

Is there a technical limit to the number of bindings that a worker can have? I'm assuming at some point there is significant performance overhead to add all of the bindings to the

env

env

object, but what order of magnitude are we talking about here?

Nnclevenger Is there a technical limit to the number of bindings that a worker can have? I'...

Hello, I’m Allie!•9/28/23, 7:28 PM

IIRC it comes down to the metadata limit, which iirc is 1 MB

Hello, I’m Allie!•9/28/23, 7:29 PM

So however many bindings you can cram into that

nclevenger•9/28/23, 7:29 PM

ahh - makes sense

nclevenger•9/28/23, 7:29 PM

I was reading the blog post today and I hadn't realized until then there was a

patch

patch

method to incrementally add a binding ... that was really interesting to see ...

Original message was deleted

Hello, I’m Allie!•9/28/23, 7:30 PM

Is that the imposed limit or technical limit?

nclevenger•9/28/23, 7:31 PM

yah - I had been using WfP as a bridge for dynamic user-specific D1 ... but one use case, I was creating the same script over and over again and I thought it would be much simpler if there was a dynamic dispatch for resources like D1

Chaika•9/28/23, 7:31 PM

The env variable limit is mentioned on https://developers.cloudflare.com/workers/platform/limits/#environment-variables, 128 for paid, 64 for free. no mention of bindings limits though, who's going to be the one to find out?

nclevenger•9/28/23, 7:32 PM

because the other issue I was running into, is as the WfP proxy was evolving, then I had to redeploy every single one of the WfP scripts, every single time it was updated

nclevenger•9/28/23, 7:32 PM

and that was turning into a technical nightmare

Nnclevenger Is there a technical limit to the number of bindings that a worker can have? I'...

Matt Silverlock•9/28/23, 7:52 PM

Each binding is (rounding up a lot) about 100-150 bytes. You can have 1MB. Includes all bindings, vars, secrets. So ballpark 5,000 (rounding down significantly).

MMatt Silverlock Each binding is (rounding up a lot) about 100-150 bytes. You can have 1MB. Inclu...

nclevenger•9/28/23, 8:27 PM

Is there added cold-start latency with a very large binding like there is with a large worker?

Nnclevenger Is there added cold-start latency with a very large binding like there is with a...

Hello, I’m Allie!•9/28/23, 8:38 PM

IIRC metadata for Workers is stored in Quicksilver, just like <1 MB Workers, so it should be fast no matter how many bindings you add

Hello, I’m Allie!•9/28/23, 8:38 PM

It’s also why Workers can be slower above 1 MB

ajgeiss0702•9/28/23, 9:09 PM

Once d1 has read replication, any idea on how consistent it would be? I’m currently planning a feature that would require semi-strong consistency

ajgeiss0702•9/28/23, 9:12 PM

(Please ping me if you reply)

Ddigeratidave what is the best practice for adding a column to an existing D1? Is the Alter co...

digeratidave•9/28/23, 10:07 PM

found I can run the alter command right from the D1 dashboard.

DDanTheGoodman I'm getting this error when using hono on a durable object that has d1 bound. No...

joel•9/28/23, 11:20 PM

Have you been able to fix it? I have the same error but im not using D1, just trying to get websockets working

Jjoel Have you been able to fix it? I have the same error but im not using D1, just tr...

DanTheGoodman•9/28/23, 11:20 PM

Haven’t touched it since

joel•9/28/23, 11:21 PM

okay :/

Adi•9/29/23, 1:19 AM

One of the biggest impediments to double-digit-gigabyte databases is performance

This line in the blog really got me thinking. I've always been using D1 with the assumption that one day this could be the primary database for the various products I'm building on Cloudflare. Reading the latest blog I somehow got the feeling that the aim is to scale D1 horizontally (aka number of databases as opposed to size of each individual database).

(Question for Matt probably) How do you envision D1 as a product? Are you aiming to build it to a point where it can be the primary database for an org? Or is D1 positioned to be on the smaller side (say up to 50GB) and powering smaller microservices in a larger architecture?

Adi•9/29/23, 1:23 AM

I suppose one way to build a primary database on D1 (assuming it scales on number of DBs as opposed to size) is some kind of sharding logic that splits the data between multiple instances. But my guess is that this kind of logic would have to come from Cloudflare, since it would be non-trivial for us to build it ourselves.

bret_pat•9/29/23, 2:49 AM

The docs say that D1 is currently in alpha

https://developers.cloudflare.com/workers/learning/storage-options/#d1

Storage options guide · Cloudflare Workers docs

The Cloudflare Workers platform provides multiple storage options. This guide will inform you on which storage option is appropriate based on your …

bret_pat•9/29/23, 3:16 AM

What RDBMS does R1 use underneath? Does R1 use clustered indexes or does it use something similar to Postgres?

Wondering about UUIDs as PK for R1

Bbret_pat The docs say that D1 is currently in alpha https://developers.cloudflare.com/wo...

Max (@rozenmd)•9/29/23, 7:01 AM

fixing that

Bbret_pat What RDBMS does R1 use underneath? Does R1 use clustered indexes or does it use...

Max (@rozenmd)•9/29/23, 7:07 AM

assuming you mean D1, it's sqlite

DDanTheGoodman I'm getting this error when using hono on a durable object that has d1 bound. No...

Matt Silverlock•9/29/23, 8:27 AM

Inside a DO? Is Hono looking for the Env in the right place? I would file a bug against Hono.

Bbret_pat The docs say that D1 is currently in alpha https://developers.cloudflare.com/wo...

Matt Silverlock•9/29/23, 8:28 AM

It’s in beta. Just missed that one page in the update.

joel•9/29/23, 8:29 AM

I somehow fixed it but have no clue how.

Steps:

Install Hono-do and add the bindings
Copy paste the example
For some reason it worked
I renamed the example class and added my code to it
I removed Hono-do again

I do not what is different than before now since it looks 1:1 the same, maybe that helps someone

joel•9/29/23, 8:29 AM

https://github.com/sor4chi/hono-do

GitHub

GitHub - sor4chi/hono-do: A wrapper of Cloudflare Workers's Durable...

A wrapper of Cloudflare Workers's Durable Object for Hono. - GitHub - sor4chi/hono-do: A wrapper of Cloudflare Workers's Durable Object for Hono.

AAdi > One of the biggest impediments to double-digit-gigabyte databases is performan...

Matt Silverlock•9/29/23, 8:56 AM

Double digits makes sense. 100GB per database is just a very different thing. But that’s also the point: orgs are used to large, monolithic databases, which makes changes and management risky. Many of us know what it’s like to have a database related incident caused by another team’s work or load.

A dominant reason everyone has big 100GB+ databases is because the cost of managing multiple database instances - the cost in operations AND dollars - is very high. You still pay by the hour. They take time to spin up. Managing backups is harder.

We think it’s going to be better for a lot of folks building new infrastructure to have 10+ 10GB D1 databases vs 1x 100GB database. And if you have legacy infrastructure, use Hyperdrive.

MMatt Silverlock Double digits makes sense. 100GB _per database_ is just a very different thing. ...

Adi•9/29/23, 9:19 AM

Thanks Matt, that makes a lot of sense. This definitely gave me a lot of clarity in how I'm thinking around future proofing products I'm building with D1

Jjoel I somehow fixed it but have no clue how. Steps: 1. Install Hono-do and add the...

Max (@rozenmd)•9/29/23, 9:30 AM

I've never heard of

hono-do

hono-do

, but the line above the install info is:

This is a work in progress. The code is not yet ready for production use.

Hono provides an example of using durable objects here: https://github.com/honojs/examples/tree/main/durable-objects

Old Man Maker•9/29/23, 9:50 AM

Probably already on a list somewhere, but would love the dashboard to show the primary key marked column

OOld Man Maker Probably already on a list somewhere, but would love the dashboard to show the p...

Max (@rozenmd)•9/29/23, 10:36 AM

it isn't actually, i'll add it to the backlog

MMatt Silverlock Inside a DO? Is Hono looking for the Env in the right place? I would file a bug ...

DanTheGoodman•9/29/23, 12:07 PM

That was like 8 months ago… I’m sure it’s changed

Max (@rozenmd)•9/29/23, 12:15 PM

Yeah at some point DOs couldn't talk to D1

DDanTheGoodman That was like 8 months ago… I’m sure it’s changed

Matt Silverlock•9/29/23, 1:51 PM

Ugh: Discord made it look like it was today

MMatt Silverlock Ugh: Discord made it look like it was today 🙃

DanTheGoodman•9/29/23, 2:07 PM

Someone replied to it yesterday

Laurent Pellegrino•9/29/23, 4:24 PM

Hello!

Is there any official docs about how to make transactions with D1? I can only find this old blog post: https://blog.cloudflare.com/whats-new-with-d1/

Laurent Pellegrino•9/29/23, 4:30 PM

How to increment a counter value stored as column in a row by 1 atomically?

Laurent Pellegrino•9/29/23, 4:30 PM

Is that supported?

Laurent Pellegrino•9/29/23, 4:48 PM

Interesting. Thanks a lot.

Laurent Pellegrino•9/29/23, 4:55 PM

Loving how D1 looks so far. I have a unique use case and need some guidance. I'm working with a database that contains roughly 50M entries. Interestingly, this database is updated only once every day, and after that, we only perform read operations. I'm curious – is there a method to either import an SQL backup or establish the database locally and then migrate it to D1, without incurring charges for the 50M writes daily? Appreciate any insights on this!

Laurent Pellegrino•9/29/23, 8:50 PM

Do you plan to keep the manual backup commands with wrangler on GA?

Cool Is time travel built on ZFS?

Similar Threads

Similar Threads

Similar Threads