> One of the biggest impediments to double-digit-gigabyte databases is performance This line in the

One of the biggest impediments to double-digit-gigabyte databases is performance

This line in the blog really got me thinking. I've always been using D1 with the assumption that one day this could be the primary database for the various products I'm building on Cloudflare. Reading the latest blog I somehow got the feeling that the aim is to scale D1 horizontally (aka number of databases as opposed to size of each individual database).

(Question for Matt probably) How do you envision D1 as a product? Are you aiming to build it to a point where it can be the primary database for an org? Or is D1 positioned to be on the smaller side (say up to 50GB) and powering smaller microservices in a larger architecture?

AdiOP•9/29/23, 1:23 AM

I suppose one way to build a primary database on D1 (assuming it scales on number of DBs as opposed to size) is some kind of sharding logic that splits the data between multiple instances. But my guess is that this kind of logic would have to come from Cloudflare, since it would be non-trivial for us to build it ourselves.

bret_pat•9/29/23, 2:49 AM

The docs say that D1 is currently in alpha

https://developers.cloudflare.com/workers/learning/storage-options/#d1

Storage options guide · Cloudflare Workers docs

The Cloudflare Workers platform provides multiple storage options. This guide will inform you on which storage option is appropriate based on your …

bret_pat•9/29/23, 3:16 AM

What RDBMS does R1 use underneath? Does R1 use clustered indexes or does it use something similar to Postgres?

Wondering about UUIDs as PK for R1

Bbret_pat The docs say that D1 is currently in alpha https://developers.cloudflare.com/wo...

Max (@rozenmd)•9/29/23, 7:01 AM

fixing that

Bbret_pat What RDBMS does R1 use underneath? Does R1 use clustered indexes or does it use...

Max (@rozenmd)•9/29/23, 7:07 AM

assuming you mean D1, it's sqlite

DDanTheGoodman I'm getting this error when using hono on a durable object that has d1 bound. No...

Matt Silverlock•9/29/23, 8:27 AM

Inside a DO? Is Hono looking for the Env in the right place? I would file a bug against Hono.

Bbret_pat The docs say that D1 is currently in alpha https://developers.cloudflare.com/wo...

Matt Silverlock•9/29/23, 8:28 AM

It’s in beta. Just missed that one page in the update.

joel•9/29/23, 8:29 AM

I somehow fixed it but have no clue how.

Steps:

Install Hono-do and add the bindings
Copy paste the example
For some reason it worked
I renamed the example class and added my code to it
I removed Hono-do again

I do not what is different than before now since it looks 1:1 the same, maybe that helps someone

joel•9/29/23, 8:29 AM

https://github.com/sor4chi/hono-do

GitHub

GitHub - sor4chi/hono-do: A wrapper of Cloudflare Workers's Durable...

A wrapper of Cloudflare Workers's Durable Object for Hono. - GitHub - sor4chi/hono-do: A wrapper of Cloudflare Workers's Durable Object for Hono.

AAdi > One of the biggest impediments to double-digit-gigabyte databases is performan...

Matt Silverlock•9/29/23, 8:56 AM

Double digits makes sense. 100GB per database is just a very different thing. But that’s also the point: orgs are used to large, monolithic databases, which makes changes and management risky. Many of us know what it’s like to have a database related incident caused by another team’s work or load.

A dominant reason everyone has big 100GB+ databases is because the cost of managing multiple database instances - the cost in operations AND dollars - is very high. You still pay by the hour. They take time to spin up. Managing backups is harder.

We think it’s going to be better for a lot of folks building new infrastructure to have 10+ 10GB D1 databases vs 1x 100GB database. And if you have legacy infrastructure, use Hyperdrive.

MMatt Silverlock Double digits makes sense. 100GB _per database_ is just a very different thing. ...

AdiOP•9/29/23, 9:19 AM

Thanks Matt, that makes a lot of sense. This definitely gave me a lot of clarity in how I'm thinking around future proofing products I'm building with D1

Jjoel I somehow fixed it but have no clue how. Steps: 1. Install Hono-do and add the...

Max (@rozenmd)•9/29/23, 9:30 AM

I've never heard of hono-dohono-do, but the line above the install info is:

This is a work in progress. The code is not yet ready for production use.

Hono provides an example of using durable objects here: https://github.com/honojs/examples/tree/main/durable-objects

Old Man Maker•9/29/23, 9:50 AM

Probably already on a list somewhere, but would love the dashboard to show the primary key marked column

OOld Man Maker Probably already on a list somewhere, but would love the dashboard to show the p...

Max (@rozenmd)•9/29/23, 10:36 AM

it isn't actually, i'll add it to the backlog

MMatt Silverlock Inside a DO? Is Hono looking for the Env in the right place? I would file a bug ...

DanTheGoodman•9/29/23, 12:07 PM

That was like 8 months ago… I’m sure it’s changed

Max (@rozenmd)•9/29/23, 12:15 PM

Yeah at some point DOs couldn't talk to D1

DDanTheGoodman That was like 8 months ago… I’m sure it’s changed

Matt Silverlock•9/29/23, 1:51 PM

Ugh: Discord made it look like it was today

MMatt Silverlock Ugh: Discord made it look like it was today 🙃

DanTheGoodman•9/29/23, 2:07 PM

Someone replied to it yesterday

Laurent Pellegrino•9/29/23, 4:24 PM

Hello!

Is there any official docs about how to make transactions with D1? I can only find this old blog post: https://blog.cloudflare.com/whats-new-with-d1/

Laurent Pellegrino•9/29/23, 4:30 PM

How to increment a counter value stored as column in a row by 1 atomically?

Laurent Pellegrino•9/29/23, 4:30 PM

Is that supported?

Laurent Pellegrino•9/29/23, 4:48 PM

Interesting. Thanks a lot.

Laurent Pellegrino•9/29/23, 4:55 PM

Loving how D1 looks so far. I have a unique use case and need some guidance. I'm working with a database that contains roughly 50M entries. Interestingly, this database is updated only once every day, and after that, we only perform read operations. I'm curious – is there a method to either import an SQL backup or establish the database locally and then migrate it to D1, without incurring charges for the 50M writes daily? Appreciate any insights on this!

Laurent Pellegrino•9/29/23, 8:50 PM

Do you plan to keep the manual backup commands with wrangler on GA?

Laurent Pellegrino•9/29/23, 8:51 PM

Does a backup restore (not using time traval) incurs a charge (i.e. does it rows as writes)?

LLaurent Pellegrino Does a backup restore (not using time traval) incurs a charge (i.e. does it rows...

Max (@rozenmd)•9/29/23, 9:13 PM

The backup commands are from a time when it was necessary to do backups with D1. Time travel lets you rollback commit by commit, so it's not as useful.

Laurent Pellegrino•9/29/23, 9:14 PM

@rozenmd I am asking because of the use case I described just above.

Laurent Pellegrino•9/29/23, 9:16 PM

I'd be willing to shell out $0.1 per million writes for my situation, but $1 per million writes is certainly off-putting.

Old Man Maker•9/30/23, 2:02 AM

Hey, can someone help me understand the D1 insert SQL a little better please.

I have an object def that looks like this:

interface DetailsDataType{
    description:string,
    ref:string //Primary Key in Table
}

interface DetailsDataType{
    description:string,
    ref:string //Primary Key in Table
}

I'm getting an array of these objects that I want to insert into my D1.

I get how to inject a single string, but the object array is not happening for me.
I assume there is some shorthand way of doing this, don't want to put an ugly for loop in there.

This is the INSERT code, where the bind statement is incorrect because descData is an array:

descData: DetailsDataType[] = <ARRAY OF OBJECTS>;

const { results } = await c.env.DATABASE.prepare("INSERT INTO details (ref, description) VALUES (?, ?) RETURNING *")
        .bind(descData.description, descData.ref)
        .run()

const { results } = await c.env.DATABASE.prepare("INSERT INTO details (ref, description) VALUES (?, ?) RETURNING *")
        .bind(descData.description, descData.ref)
        .run()

Can this even be done this way?

Old Man Maker•9/30/23, 2:24 AM

I'm assuming I will need to use the Batching statement?

Old Man Maker•9/30/23, 2:42 AM

Ok - just got this to work with a looping batcher. Pretty sweet.

Can I make a suggestion please - if the use of hyphens '-' in names is causing so many issues, why not just blanket ban the use in any named object.

It just took me 10 mins to spot the issue with the table name, when it should have been rejected as a valid table name at the point of creation. Perhaps the wrangler tool does this, but the dashboard does not.

Thoughts?

franz101•9/30/23, 11:34 AM

What's the best practices for an ETL to export data from D1?

Ffranz101 What's the best practices for an ETL to export data from D1?

Dahoom•9/30/23, 11:37 AM

For the beta I'm currenly using community member nora's repo https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

Dahoom•9/30/23, 11:38 AM

it works pretty well but I dont think there is any official system the beta system yet

franz101•9/30/23, 11:44 AM

Ah interesting I saw the dump function. I was curious as I was wondering about a local python connection to query the db. I guess you would need a worker that evals the query. I'm testing d1 for my ai news aggregator:
https://ml-news-liard.vercel.app/

Ffranz101 Ah interesting I saw the dump function. I was curious as I was wondering about a...

Dahoom•9/30/23, 11:49 AM

You probably need a worker yeah, I know for a fact you can connect directly to turso currently with libsql

Jason Hostetter•9/30/23, 3:25 PM

Is it correct that there is no way to programmatically create or connect to different databases for users without an enterprise account?

JJason Hostetter Is it correct that there is no way to programmatically create or connect to diff...

Cyb3r-Jak3•9/30/23, 3:30 PM

I'm pretty sure even with enterprise you can't

Jason Hostetter•9/30/23, 3:32 PM

There was mention of creating one db per client in the beta announcement, but only for enterprise

Cyb3r-Jak3•9/30/23, 3:32 PM

Oh workers for platforms

Jason Hostetter•9/30/23, 3:32 PM

Although the description wasn’t super clear to me

Jason Hostetter•9/30/23, 3:32 PM

Yeah

Cyb3r-Jak3•9/30/23, 3:33 PM

Yeah, sorry haven't used WfP but that will allow you to

Jason Hostetter•9/30/23, 3:34 PM

Is workers for platforms enterprise only?

Cyb3r-Jak3•9/30/23, 3:35 PM

It is

Jason Hostetter•9/30/23, 3:35 PM

Bahhh

Jason Hostetter•9/30/23, 3:38 PM

Ok well, maybe I’ll have to try to hack something together with turso or something

> One of the biggest impediments to double-digit-gigabyte databases is performance This line in the

Similar Threads

Similar Threads

Similar Threads