Cloudflare Developers•3y ago

Not at the moment. Prioritizing larger DBs, performance, and bringing branching & cloning to Time Tr

Not at the moment. Prioritizing larger DBs, performance, and bringing branching & cloning to Time Travel first.

MMatt Silverlock Not at the moment. Prioritizing larger DBs, performance, and bringing branching ...

Sameer Ali•7/30/23, 8:37 PM

First one is the most exciting and much needed! Larger DBs!

MMatt Silverlock Not at the moment. Prioritizing larger DBs, performance, and bringing branching ...

Bram•7/30/23, 9:22 PM

question on the "larger databases" part, is the plan to (eventually) to support large databases of 100GB - 500GB?

Unsmart•7/30/23, 9:25 PM

This was the last message in regards to DB size and based on it I'd assume under 100GB d1-database

Unsmart•7/30/23, 9:28 PM

(also scroll down a bit for the talk about adding some form of way for you to shard a db)

Bram•7/30/23, 9:39 PM

thx, sharding would be completely reasonable of course

λ.ςζ•7/31/23, 4:04 PM

is it possible to get a dump of D1 (alpha) database?

λ.ςζ•7/31/23, 4:09 PM

could you point me how? i couldn't find anything helpful

λ.ςζ•7/31/23, 4:11 PM

oh. i've tried that.

Experimental DBs do not support backups yet

Experimental DBs do not support backups yet

Cyb3r-Jak3•7/31/23, 4:14 PM

It’s still alpha just the new vs old backend. The new backend doesn’t support direct dumps but rather time travel

λ.ςζ•7/31/23, 4:15 PM

thanks

ZZiga Zajc So both old backend and new backend are still called Alpha?

DaniFoldi•7/31/23, 4:17 PM

correct

Hyper•7/31/23, 7:58 PM

I saw somewhere on the docs that running wrangler d1 info or a command that is similar gives u either version: alpha or version: beta based on whether you’re using the exp backend or not

HHyper I saw somewhere on the docs that running wrangler d1 info or a command that is s...

Matt SilverlockOP•7/31/23, 8:32 PM

Right at the top of https://developers.cloudflare.com/d1/learning/time-travel/

MMatt Silverlock Right at the top of https://developers.cloudflare.com/d1/learning/time-travel/

Cyb3r-Jak3•7/31/23, 8:33 PM

I think the confusion is because d1 says open alpha

CCyb3r-Jak3 I think the confusion is because d1 says open alpha

Matt SilverlockOP•7/31/23, 8:34 PM

Choice we had to make. This will be the backend going forward.

MMatt Silverlock Choice we had to make. This will be the backend going forward.

Cyb3r-Jak3•7/31/23, 8:36 PM

Yeah I figured the beta backend will be the one for open beta just confusing for now

chientrm.com•7/31/23, 10:35 PM

Cloudflare: D1 is currently in Open Alpha and is not recommended for production data and trafficD1 is currently in Open Alpha and is not recommended for production data and traffic
Me: Move database from PlanetScale to D1Move database from PlanetScale to D1

chientrm.com•7/31/23, 10:36 PM

I don't see the option to use new D1 backend, is it the default now?

James•7/31/23, 10:46 PM

It's default now, yes! wrangler d1 listwrangler d1 list should give you a list back of your D1 databases and what backend they're running on. If you see version: betaversion: beta, they're on the new backend

Cchientrm.com I don't see the option to use new D1 backend, is it the default now? 🤔

Matt SilverlockOP•7/31/23, 10:50 PM

https://developers.cloudflare.com/d1/changelog/#new-default-storage-subsystem

Changelog · Cloudflare D1 docs

Databases created via the Cloudflare dashboard and Wrangler (as of v3.4.0) now use D1’s new storage subsystem by default. The new backend can be 6 - …

Hello, I’m Allie!•7/31/23, 11:17 PM

Given a CSV file containing data bound for a D1 table, what is the most optimal way to update said table, reducing reads/writes?

Hello, I’m Allie!•7/31/23, 11:18 PM

I don’t know ahead of time what has changed, nor even if anything has changed

TigersWay•7/31/23, 11:46 PM

Following some previous comments, should we conclude that the API dumpdump is not working yet/anymore on the new/default/experimental backend... Am I right?

chientrm.com•7/31/23, 11:56 PM

using D1, how to get rowsEffectedrowsEffected?

HHello, I’m Allie!Given a CSV file containing data bound for a D1 table, what is the most optimal ...

Matt SilverlockOP•8/1/23, 12:20 AM

You’d have to read it to know what changed. Sounds like premature optimization? How many rows is the CSV that this matters?

TTigersWay Following some previous comments, should we conclude that the API `dump` is not ...

Matt SilverlockOP•8/1/23, 12:21 AM

Correct, per the docs: https://developers.cloudflare.com/d1/learning/backups/

there are not near term plans to support dump.

Backups (Legacy) · Cloudflare D1 docs

D1 has built-in support for creating and restoring backups of your databases, including support for scheduled automatic backups and manual backup …

TTigersWay Following some previous comments, should we conclude that the API `dump` is not ...

Matt SilverlockOP•8/1/23, 12:21 AM

d1-database

Time Travel is much more powerful.

MMatt Silverlock https://discord.com/channels/595317990191398933/992060581832032316/1135238676264...

TigersWay•8/1/23, 12:23 AM

But right now I have no way to "recover" a DB full of real data!
I guess I'm gonna start and write an export API

MMatt Silverlock You’d have to read it to know what changed. Sounds like premature optimization? ...

Hello, I’m Allie!•8/1/23, 12:52 AM

One of the files in question has around ~68k rows. And I could probably just run it as a single/few large

UPDATE

UPDATE

s, but that seems pretty slow...

Hello, I’m Allie!•8/1/23, 12:54 AM

And in case I do need to run each separately, would it be more efficient to run it as a few large statements, or each row as a single statement in a batch?

Hello, I’m Allie!•8/1/23, 12:57 AM

To be clear, I'm not expecting to be able to do it within a single read/write, I'm just wondering if there is anything I could do that is faster/more efficient than ~70k+ inserts every day...

Hello, I’m Allie!•8/1/23, 1:00 AM

If it matters, my dataset is a GTFS dump from BART. In this case, the number of rows is comparatively small, but it can scale to a fair bit large with larger transit systems, so I'm just going for best practices here

Hello, I’m Allie!•8/1/23, 1:03 AM

And it shouldn't

be outside the workable range of D1, as the CSV-form dataset is currently 6.5 MB

HHello, I’m Allie!One of the files in question has around ~68k rows. And I could probably just run...

Matt SilverlockOP•8/1/23, 1:06 AM

Or https://www.sqlite.org/lang_UPSERT.html

MMatt Silverlock Or https://www.sqlite.org/lang_UPSERT.html - ?

Hello, I’m Allie!•8/1/23, 3:08 AM

Would there also be a handy way to drop rows that aren’t in the CSV dataset?

Not at the moment. Prioritizing larger DBs, performance, and bringing branching & cloning to Time Tr

Similar Threads

Similar Threads

Similar Threads