Is support for a large number of d1 instances going to be solely tied to Workers for Platforms users

Is support for a large number of d1 instances going to be solely tied to Workers for Platforms users (which needs ent contract), or is it something which would be generally available?

EEpailes Is support for a large number of d1 instances going to be solely tied to Workers...

Matt Silverlock•9/28/23, 1:35 PM

We can increase on a case-by-case basis. We intend to substantially increase the # of databases per account (per the blog) during the beta period as well.

EEpailes Is support for a large number of d1 instances going to be solely tied to Workers...

Matt Silverlock•9/28/23, 1:36 PM

At "100k databases" there are other things that make you want to use WfP in terms of how many bindings you can have per script, how you are managing user scripts, etc.

MMatt Silverlock At "100k databases" there are other things that make you want to use WfP in term...

EpailesOP•9/28/23, 1:38 PM

Ah I thought if D1 supported a large number of instances it would move to the DO method of connecting where you don't need an individual binding per worker

EpailesOP•9/28/23, 1:40 PM

At least for the use case where each instance has the same schema and many instances is more a method of sharding to prevent everything being stored in one huge instance which slows down, and there's no legal/contractual requirements to store user data separately.
If each d1 can be unique then platforms makes a lot more sense

EEpailes Ah I thought if D1 supported a large number of instances it would move to the DO...

Matt Silverlock•9/28/23, 1:43 PM

Something we're working on (search this channel) but not yet.

Original message was deleted

Jacob Wright•9/28/23, 1:56 PM

I saw Cloudflare added Turso as an integration partner and was wondering the pros/cons of Turso vs D1 in your experience

Ashish Rawat•9/28/23, 2:04 PM

How this will work for 1000s of users if the database limit will be in GB (assuming atmost 10GB in future) with just 25 database per project.

Mozzy•9/28/23, 2:17 PM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

Is this Responses documentation wrong?
I'd expect it to return an array of rows

AAshish Rawat How this will work for 1000s of users if the database limit will be in GB (assum...

Matt Silverlock•9/28/23, 2:20 PM

Per database

Adi•9/28/23, 2:22 PM

Seriously a huge congratulations to the D1 team for hitting beta status

I can't wait to see everything you do in the future

MMozzy <https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database> ...

Max (@rozenmd)•9/28/23, 2:40 PM

My mistake, will fix

Rryoppippi hi is d1 production ready?

Max (@rozenmd)•9/28/23, 3:10 PM

D1 is in public beta. While we expect breaking changes and issues to be minimal, they may still occur. We generally do not recommend running large production workloads on beta products.

https://developers.cloudflare.com/d1/

MMax (@rozenmd)My mistake, will fix

Max (@rozenmd)•9/28/23, 3:54 PM

fixed

PizzaConsole•9/28/23, 4:28 PM

I see that D1 does not support Transactions or save points. However when using a transaction and running locally it does not throw an error. Is that a change that needs to happen in Miniflare?

LLogan Grasby Exciting updates! If you have a D1 database already is there anything we need to...

Max (@rozenmd)•9/28/23, 4:31 PM

nope, as long as your db says "beta" when you run wrangler d1 info, you're all set

Jacob Wright•9/28/23, 5:26 PM

Can I update my alpha database to become beta, or do I need to create a new database? And if the latter, will I need to do the same for general release?

JJacob Wright Can I update my alpha database to become beta, or do I need to create a new data...

Isaac McFadyen•9/28/23, 5:32 PM

You'll need to dump the database to a SQLite file and then re-import to a new (beta) database.

IIsaac McFadyen You'll need to dump the database to a SQLite file and then re-import to a new (b...

Jacob Wright•9/28/23, 5:34 PM

Will we need to do the same at general release?

JJacob Wright Will we need to do the same at general release?

Isaac McFadyen•9/28/23, 5:37 PM

I'm unsure, and I doubt anyone will be able to tell you since nothing is concrete until GA.

abdellah•9/28/23, 5:51 PM

Cool
Is time travel built on ZFS?

IIsaac McFadyen I'm unsure, and I doubt anyone will be able to tell you since nothing is concret...

Jacob Wright•9/28/23, 6:14 PM

Good point.

JJacob Wright Will we need to do the same at general release?

Matt Silverlock•9/28/23, 7:13 PM

Not if you are on the beta backend.

Aabdellah Cool Is time travel built on ZFS?

Matt Silverlock•9/28/23, 7:14 PM

No. ZFS snapshots don’t work like that (and way too low level). D1 has its own storage subsystem that allows this to work.

MMatt Silverlock At "100k databases" there are other things that make you want to use WfP in term...

nclevenger•9/28/23, 7:27 PM

Is there a technical limit to the number of bindings that a worker can have? I'm assuming at some point there is significant performance overhead to add all of the bindings to the

env

env

object, but what order of magnitude are we talking about here?

Nnclevenger Is there a technical limit to the number of bindings that a worker can have? I'...

Hello, I’m Allie!•9/28/23, 7:28 PM

IIRC it comes down to the metadata limit, which iirc is 1 MB

Hello, I’m Allie!•9/28/23, 7:29 PM

So however many bindings you can cram into that

nclevenger•9/28/23, 7:29 PM

ahh - makes sense

nclevenger•9/28/23, 7:29 PM

I was reading the blog post today and I hadn't realized until then there was a patchpatch method to incrementally add a binding ... that was really interesting to see ...

Original message was deleted

Hello, I’m Allie!•9/28/23, 7:30 PM

Is that the imposed limit or technical limit?

nclevenger•9/28/23, 7:31 PM

yah - I had been using WfP as a bridge for dynamic user-specific D1 ... but one use case, I was creating the same script over and over again and I thought it would be much simpler if there was a dynamic dispatch for resources like D1

Chaika•9/28/23, 7:31 PM

The env variable limit is mentioned on https://developers.cloudflare.com/workers/platform/limits/#environment-variables, 128 for paid, 64 for free. no mention of bindings limits though, who's going to be the one to find out?

nclevenger•9/28/23, 7:32 PM

because the other issue I was running into, is as the WfP proxy was evolving, then I had to redeploy every single one of the WfP scripts, every single time it was updated

nclevenger•9/28/23, 7:32 PM

and that was turning into a technical nightmare

Nnclevenger Is there a technical limit to the number of bindings that a worker can have? I'...

Matt Silverlock•9/28/23, 7:52 PM

Each binding is (rounding up a lot) about 100-150 bytes. You can have 1MB. Includes all bindings, vars, secrets. So ballpark 5,000 (rounding down significantly).

MMatt Silverlock Each binding is (rounding up a lot) about 100-150 bytes. You can have 1MB. Inclu...

nclevenger•9/28/23, 8:27 PM

Is there added cold-start latency with a very large binding like there is with a large worker?

Nnclevenger Is there added cold-start latency with a very large binding like there is with a...

Hello, I’m Allie!•9/28/23, 8:38 PM

IIRC metadata for Workers is stored in Quicksilver, just like <1 MB Workers, so it should be fast no matter how many bindings you add

Hello, I’m Allie!•9/28/23, 8:38 PM

It’s also why Workers can be slower above 1 MB

ajgeiss0702•9/28/23, 9:09 PM

Once d1 has read replication, any idea on how consistent it would be? I’m currently planning a feature that would require semi-strong consistency

ajgeiss0702•9/28/23, 9:12 PM

(Please ping me if you reply)

Ddigeratidave what is the best practice for adding a column to an existing D1? Is the Alter co...

digeratidave•9/28/23, 10:07 PM

found I can run the alter command right from the D1 dashboard.

DDanTheGoodman I'm getting this error when using hono on a durable object that has d1 bound. No...

joel•9/28/23, 11:20 PM

Have you been able to fix it? I have the same error but im not using D1, just trying to get websockets working

Jjoel Have you been able to fix it? I have the same error but im not using D1, just tr...

DanTheGoodman•9/28/23, 11:20 PM

Haven’t touched it since

joel•9/28/23, 11:21 PM

okay :/

Adi•9/29/23, 1:19 AM

One of the biggest impediments to double-digit-gigabyte databases is performance

This line in the blog really got me thinking. I've always been using D1 with the assumption that one day this could be the primary database for the various products I'm building on Cloudflare. Reading the latest blog I somehow got the feeling that the aim is to scale D1 horizontally (aka number of databases as opposed to size of each individual database).

(Question for Matt probably) How do you envision D1 as a product? Are you aiming to build it to a point where it can be the primary database for an org? Or is D1 positioned to be on the smaller side (say up to 50GB) and powering smaller microservices in a larger architecture?

Adi•9/29/23, 1:23 AM

I suppose one way to build a primary database on D1 (assuming it scales on number of DBs as opposed to size) is some kind of sharding logic that splits the data between multiple instances. But my guess is that this kind of logic would have to come from Cloudflare, since it would be non-trivial for us to build it ourselves.

bret_pat•9/29/23, 2:49 AM

The docs say that D1 is currently in alpha

https://developers.cloudflare.com/workers/learning/storage-options/#d1

Storage options guide · Cloudflare Workers docs

The Cloudflare Workers platform provides multiple storage options. This guide will inform you on which storage option is appropriate based on your …

Is support for a large number of d1 instances going to be solely tied to Workers for Platforms users

Similar Threads

Similar Threads

Similar Threads