Cloudflare Developers•3y ago

Got it, that's good to know 🙂 . I suppose once backups are available it makes sense to switch then

Got it, that's good to know

. I suppose once backups are available it makes sense to switch then

AAdi Got it, that's good to know 🙂 . I suppose once backups are available it makes s...

Cyb3r-Jak3•6/29/23, 1:36 AM

Keep in mind it isn't going to be a sql dumb zip but Time Travel.
https://blog.cloudflare.com/d1-turning-it-up-to-11/ search for Time Travel header

pl•6/29/23, 8:23 AM

i have a decent oltp db in mysql, about 150gb, that i would love to get into d1 so the performance is going to be a real issue

pl•6/29/23, 10:37 AM

i have workers serving 500 req/sec out of kv at the moment but im really interested in having the backend db in d1, pushing aggregated data into kv

Max (@rozenmd)•6/29/23, 11:16 AM

fwiw #analytics-engine-beta is ridiculously good for high rps analytics data, we're using it to power the stats in the dashboard

Mirza A. Marotsaha•6/29/23, 2:28 PM

Hello, I planned to use d1 on a future product. I'm asking if there will be some breaking change from the alpha release to the production ready d1 ?

Unsmart•6/29/23, 2:33 PM

It is possible

Unsmart•6/29/23, 9:25 PM

d1-database

Matt Silverlock•6/29/23, 10:16 PM

Not Final / Initial Release / MVP but... we're close. Hoping to land this within the next 2 weeks.

Our v1.0 will replace the DB in place.
In future, we'll default to a --fork/--clone --fork/--clone approach and make overwriting (even more) explicit.
We'll explore adding helper flags - e.g. --minutes-ago=15m--minutes-ago=15m or similar to avoid you having to deal with Unix timestamps.

MMatt Silverlock Not Final / Initial Release / MVP but... we're close. Hoping to land this within...

itsmatteomanf•6/29/23, 10:19 PM

Ideally it'd be great to have either a duplicate or a way to lock a version too keep it "forever".

Iitsmatteomanf Ideally it'd be great to have either a duplicate or a way to lock a version too ...

Matt Silverlock•6/29/23, 10:20 PM

Right, that'd be --clone/--fork--clone/--fork

just copy it to a fresh DB and do what you want. --clone=new-name--clone=new-name

MMatt Silverlock Right, that'd be `--clone/--fork` - just copy it to a fresh DB and do what you w...

itsmatteomanf•6/29/23, 10:21 PM

The "lock" action is very intuitive given it's common for many things, but this would be great regardless.

Iitsmatteomanf The "lock" action is very intuitive given it's common for many things, but this ...

Matt Silverlock•6/29/23, 10:23 PM

Can you expand on what "lock" means - e.g. no longer accepting writes? At all?

MMatt Silverlock Can you expand on what "lock" means - e.g. no longer accepting writes? At all?

itsmatteomanf•6/29/23, 10:24 PM

Let me find you an example.

Matt Silverlock•6/29/23, 10:24 PM

Or do you want to say "keep this version forever" (which is... cloning that bookmark to a DB you can keep)

Matt Silverlock•6/29/23, 10:25 PM

(Then you end up with several things to manage - you have "regular" databases and then some set of "persistent locked copies" that exist at a meta layer, and charge you for storage)

MMatt Silverlock Or do you want to say "keep this version forever" (which is... cloning that book...

itsmatteomanf•6/29/23, 10:25 PM

I meant this yes, I get it's the same, I'd be fine regardless... but it's way easier UX wise, less duplicate things and it's all compressed into one entry.

Iitsmatteomanf I meant this yes, I get it's the same, I'd be fine regardless... but it's way ea...

Matt Silverlock•6/29/23, 10:25 PM

See what I wrote above. I don't agree. We're gonna charge you to keep it around "forever", sorry.

MMatt Silverlock See what I wrote above. I don't agree. We're gonna charge you to keep it around ...

itsmatteomanf•6/29/23, 10:26 PM

Oh yeah, not expecting you to not charge for that

itsmatteomanf•6/29/23, 10:26 PM

Definitely do, everyone does.

Matt Silverlock•6/29/23, 10:26 PM

And then if you want to query that "locked" version, is that some special binding? How that does map to just being "another" DB? etc.

itsmatteomanf•6/29/23, 10:27 PM

Oh no no, it's not an actually present and accessible db. It'd be a sort of snapshot saved you can revert to or clone out of.

itsmatteomanf•6/29/23, 10:27 PM

You could use it to create another db or revert the one you are using back to that.

Matt Silverlock•6/29/23, 10:29 PM

(I'm intentionally being a little obtuse because I'm trying to understand why adding "more" to the data model is helpful) - so why not just make that another DB

Matt Silverlock•6/29/23, 10:29 PM

If we let you have ~100+ DBs per account, or 1000?

itsmatteomanf•6/29/23, 10:30 PM

Scaleway, for example, in their DB implementation (it's somewhat different, those are actual VMs running a managed DB instance on them) has a backup/snapshot feature. Billed per GB/month (sort of like it being saved to an object storage) which get discarded after some time you define. You can lock version to bypass this and keep them forever (with the cost, of course).

Matt Silverlock•6/29/23, 10:30 PM

Right, but that makes sense when you think about disk snapshots in a traditional, VM based system, without point in time recovery.

Matt Silverlock•6/29/23, 10:30 PM

So you have to take snapshots.

MMatt Silverlock (I'm intentionally being a little obtuse because I'm trying to understand why ad...

itsmatteomanf•6/29/23, 10:32 PM

Because, let's imagine you have a sort of sample dataset you use. You have multiple Workers, Pages sites, etc. connected to it. Now, you clone it out for initial data. You run things for a while and then you want to revert. Now you have to go and change where each of those things point to (by cloning the "backup" to a separate thing, because otherwise you loose it).

itsmatteomanf•6/29/23, 10:32 PM

With a locked "snapshot" you can define, you just go there, click and replace the current instance.

itsmatteomanf•6/29/23, 10:32 PM

Or create another one.

MMatt Silverlock If we let you have ~100+ DBs per account, or 1000?

itsmatteomanf•6/29/23, 10:33 PM

Oh yeah, again, I'd be fine with both instances, they'd both work fine. It's mostly UX and having users handle the duplication vs. a simple "lock" which handles things in the backed automatically.

Matt Silverlock•6/29/23, 10:33 PM

I don't think that requires snapshots. If we let you clone over the top of another DB... then you can achieve that.

itsmatteomanf•6/29/23, 10:34 PM

Oh if you do, then ok. Then 99% of the thing goes away.

Matt Silverlock•6/29/23, 10:34 PM

The more new concepts we add, and the more "things" that can eat your bill (we have to list your snapshots, show usage by snapshot, etc) -> way more complexity.

itsmatteomanf•6/29/23, 10:34 PM

I assumed you could only clone to new or to itself

Matt Silverlock•6/29/23, 10:34 PM

This is why I push you

Matt Silverlock•6/29/23, 10:34 PM

Now, you clone it out for initial data. You run things for a while and then you want to revert. Now you have to go and change where each of those things point to (by cloning the "backup" to a separate thing, because otherwise you loose it).

itsmatteomanf•6/29/23, 10:34 PM

Oh yeah, I'm happy to engage in this

Matt Silverlock•6/29/23, 10:34 PM

That was your problem in a nutshell

MMatt Silverlock That was your problem in a nutshell 😉

itsmatteomanf•6/29/23, 10:36 PM

Yeah, with the ability to clone over something else that exists already then yeah. it's just some UX, but it makes sense. Cloudflare is more dev focused, a Scaleway has a slightly wider user base that would go to it.

MMatt Silverlock Yes - 100MB is the limit for now. We aren’t increasing database limits _yet_. Ho...

Victor•6/30/23, 5:47 PM

@elithrar Do you guys have a date in mind when you will start allowing size limit increase?

VVictor @elithrar Do you guys have a date in mind when you will start allowing size limi...

Matt Silverlock•6/30/23, 5:49 PM

Not that we're sharing yet. It will coincide with the beta. We won't be doing one-off increases.

Sameer Ali•6/30/23, 7:25 PM

@elithrar D1 Seems pretty stable now, what's holding off beta other than metrics and time-travel?

Victor•6/30/23, 8:50 PM

In https://developers.cloudflare.com/workers/wrangler/configuration/#d1-databases, it says to use idid (in line with kv format), but then the example uses database_iddatabase_id. Which one is it?

Configuration · Cloudflare Workers docs

Wrangler optionally uses a wrangler.toml configuration file to customize the development and publishing setup for a Worker.

VVictor In https://developers.cloudflare.com/workers/wrangler/configuration/#d1-database...

Cyb3r-Jak3•6/30/23, 8:51 PM

database_iddatabase_id

chenyh_henry•7/1/23, 9:25 AM

Hi everyone, I face an issue or not support that the id SERIAL PRIMARY KEY id SERIAL PRIMARY KEY seems not work, I follow the doc https://developers.cloudflare.com/workers/tutorials/postgres/#5-interact-with-the-products-database, I know the condition is own host PostgreSQL but the D1 should be the same, right?

From the screenshot, the ID is always null.

Connect to a PostgreSQL database with Cloudflare Workers · Cloudfla...

In this tutorial, you will learn how to create a Cloudflare Workers application and connect it to a PostgreSQL database using TCP Sockets. The Workers …

vassbence•7/1/23, 9:33 AM

D1 is running on SQLite, not PostgreSQL

vassbence•7/1/23, 9:33 AM

try id int primary key autoincrementid int primary key autoincrement

Got it, that's good to know 🙂 . I suppose once backups are available it makes sense to switch then

Similar Threads