Hi, I just published a new package to allow creating backups of D1 databases without using a Worker.

Hi, I just published a new package to allow creating backups of D1 databases without using a Worker. Hopefully this helps someone!
https://github.com/Cretezy/cloudflare-d1-backup

GitHub

GitHub - Cretezy/cloudflare-d1-backup: Dump Cloudflare D1 databases...

Dump Cloudflare D1 databases (without Workers). Contribute to Cretezy/cloudflare-d1-backup development by creating an account on GitHub.

JSON•10/10/23, 5:51 AM

I'm asking you again because d1 is beta.
Do I still have to do the vacuum command manually?

CretezyOP•10/10/23, 6:11 AM

Thank you!

Max (@rozenmd)•10/10/23, 7:50 AM

i think i might just add something like that to wrangler

Max (@rozenmd)•10/10/23, 7:51 AM

we'll eventually rewrite it to use workerd but running select * ...select * ... against your whole db and dumping to a local sql file i can ship today

CCretezy Hi, I just published a new package to allow creating backups of D1 databases wit...

w3bcode•10/10/23, 8:11 AM

Thanks a lot for this

MMax (@rozenmd)i think i might just add something like that to wrangler

w3bcode•10/10/23, 8:13 AM

I would really appreciate a way to backup d1 database locally. I pitched D1 storage but as a fail safe, the client didnt trust enough on time travel and wanted a way to take backup locally on disk.

Paolo•10/10/23, 12:31 PM

does sequelize work with D1? can you suggest another ORM by any chance?

PPaolo does sequelize work with D1? can you suggest another ORM by any chance?

DaniFoldi•10/10/23, 12:41 PM

I recommend drizzle, we recently started migrating all projects to it and it works flawlessly - with D1, with postgres (through hyperdrive), and others

Ddpeek I'm seeing some performance numbers with D1 that don't add up... wondering if th...

Max (@rozenmd)•10/10/23, 1:19 PM

what you're seeing is the difference between how long d1 took to execute your query, and how long it took to go from request -> query -> response

Max (@rozenmd)•10/10/23, 1:19 PM

we return just the query response time, since that's controllable by the user (request less data/index your tables etc)

DDaniFoldi I recommend drizzle, we recently started migrating all projects to it and it wor...

Michał•10/10/23, 1:28 PM

Did you observer any performance penalty when using Drizzle with D1 compared to just raw SQL and prepared statements?

MMichał Did you observer any performance penalty when using Drizzle with D1 compared to ...

DaniFoldi•10/10/23, 1:29 PM

nothing measurable, so at most a couple ms - and the queries drizzle generates are now pretty efficient too, so in some cases it might be better on the DB site if we don't hand-write everythign

GuillaumeLakano•10/10/23, 3:19 PM

Hi!
An assigned UUID to a D1 database will stay the same forever, even if we move from Beta to GA, or if we need to uses Time Travel ?
To give some context, I'm asking because as the concept of database-per-user use-cases will be soon possible (only manual bindings in wrangler.toml for now and limited to ~5K), I suppose the future dynamic binding will uses the UUID assigned to the D1 database. Then, we want to save this UUID in other databases, and want to be sure this UUID will never change later, we could even consider it as a replacement of a classic id_customer field.

GGuillaumeLakano Hi! An assigned UUID to a D1 database will stay the same forever, even if we mo...

Isaac McFadyen•10/10/23, 3:21 PM

Time travel = yes, stable UUID
Alpha -> Beta, not stable, because you need to recreate the database
Beta -> GA, probably stable, an employee has said here before that there shouldn't need to be any migrations involving recreating of databases. Ref: https://canary.discord.com/channels/595317990191398933/992060581832032316/1153997053861576744

Isaac McFadyen•10/10/23, 3:21 PM

But D1 is in beta and no guarantees are made. If something goes wrong or DBs need to be migrated the UUID may change.

IIsaac McFadyen Time travel = yes, stable UUID Alpha -> Beta, not stable, because you need to re...

GuillaumeLakano•10/10/23, 3:22 PM

Thank you for your feedback !

MrXO•10/10/23, 3:45 PM

I am working on python application - can we use python connect d1 db?

MMrXO I am working on python application - can we use python connect d1 db?

Steved•10/10/23, 4:18 PM

Probably though the REST api. https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

SSteved Probably though the REST api. https://developers.cloudflare.com/api/operations/c...

MrXO•10/10/23, 4:24 PM

nice thank you

MMrXO I am working on python application - can we use python connect d1 db?

Max (@rozenmd)•10/10/23, 4:42 PM

If you need performance, write a Worker then talk to it over HTTP from python

MMax (@rozenmd)If you need performance, write a Worker then talk to it over HTTP from python

Matt Silverlock•10/10/23, 6:03 PM

https://github.com/elithrar/http-api-d1-example

GitHub

GitHub - elithrar/http-api-d1-example: An example HTTP API for Clou...

An example HTTP API for Cloudflare's D1 database: https://developers.cloudflare.com/d1/

GitHub - elithrar/http-api-d1-example: An example HTTP API for Cloudflare's D1 database: ht...

andyt•10/10/23, 6:55 PM

Hi, I am running wrangler pages dev --compatibility-date=2023-01-01 ./pages --kv=KV --d1=DBwrangler pages dev --compatibility-date=2023-01-01 ./pages --kv=KV --d1=DB and it cannot find my table, I have assumed the d1 binding is to my remote database? is this not the case?

andyt•10/10/23, 6:58 PM

ah ok, how do you create a local d1 database and script it?

CretezyOP•10/10/23, 9:05 PM

Why is the D1 HTTP API lower performance than the API in Workers? (excluding that it's not in the same datacenter)

Original message was deleted

CretezyOP•10/10/23, 9:10 PM

That's the "(excluding that it's not in the same datacenter)" part, but it seems like the API is quite slow, even for remote

CretezyOP•10/10/23, 9:12 PM

I'm seeing 500-700ms for simple queries

Max (@rozenmd)•10/10/23, 9:12 PM

The API isn't designed for Workers-style access patterns, it's an API

Max (@rozenmd)•10/10/23, 9:13 PM

whereas a Worker will be within 50ms of 95% of the internet-connected world

CretezyOP•10/10/23, 9:15 PM

But why isn't it designed for this? Multiple queries in a Worker is <100ms, a single of those queries outside is >500ms. They are designed for Workers, but the API being 5x slower seems like a missed opportunity

CretezyOP•10/10/23, 9:15 PM

The alternative is making a query worker such as http-api-d1-examplehttp-api-d1-example above, but it feels like the API should be able to do that

Original message was deleted

itsmatteomanf•10/10/23, 9:27 PM

This, and also the core datacenter might be distant from the D1 location, which adds to the roundtrip time.

itsmatteomanf•10/10/23, 9:27 PM

You are generally close to the database you create, so it's fast you -> worker -> db.

itsmatteomanf•10/10/23, 9:28 PM

If you use the API you do you -> api (possibly far away) -> worker (it's an assumption, but I bet it's a worker or similar) -> db (close to you).

itsmatteomanf•10/10/23, 9:29 PM

This makes queries at least 4x the time to go from you to the API, plus inefficiencies as connections need to be established.

Iitsmatteomanf This, and also the core datacenter might be distant from the D1 location, which ...

Max (@rozenmd)•10/10/23, 9:42 PM

this, we built an API rapidly with what we already had, we might evolve it in the future but that's what's there today

Paul•10/10/23, 9:46 PM

Any guis work with D1 for viewing live data?

Paul•10/10/23, 9:47 PM

(It seems I can use dbeaver for viewing the local SQLite file)

PPaul Any guis work with D1 for viewing live data?

Cyb3r-Jak3•10/10/23, 9:57 PM

There is a console in the dashboard

CCyb3r-Jak3 There is a console in the dashboard

Paul•10/10/23, 10:01 PM

Are you able to run arbitrary queries from the dashboard?

PPaul Are you able to run arbitrary queries from the dashboard?

Cyb3r-Jak3•10/10/23, 10:01 PM

Yeah. Go to your D1 database in the dashboard and there will a tab of ConsoleConsole

Paul•10/10/23, 10:19 PM

Ok great. Is there a plan to support more traditional non web based clients

PPaul Ok great. Is there a plan to support more traditional non web based clients

Cyb3r-Jak3•10/10/23, 10:24 PM

Personally, I wouldn't expect any official support given that D1 would only be queryable over HTTP but people from the community might add support to the various projects

CCyb3r-Jak3 Personally, I wouldn't expect any official support given that D1 would only be q...

Hello, I’m Allie!•10/10/23, 10:43 PM

Is HTTP even officially supported? I know it is wrangler, but I forget if it is documented

kian•10/10/23, 10:44 PM

Yeah

kian•10/10/23, 10:44 PM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

Cyb3r-Jak3•10/10/23, 10:54 PM

^ Came with upgrade to beta

ChrisBrocklesby•10/10/23, 10:55 PM

Hi, I am having major issues uploading a large amount of data from a .sql dump to D1 "136,000 rows" any tips / help you guys can offer ?

CChrisBrocklesby Hi, I am having major issues uploading a large amount of data from a .sql dump t...

Cyb3r-Jak3•10/10/23, 11:00 PM

You could try splitting the sql dump file into multiple files and do each one separately

Hi, I just published a new package to allow creating backups of D1 databases without using a Worker.

Similar Threads

Similar Threads

Similar Threads