Saw you guys made d1 open beta, did you fix the issue where it'll go down for 20+ hours?

EvanPiro•10/6/23, 4:53 PM

I'm getting the following error:

$ npx wrangler d1 backup create prod-db$ npx wrangler d1 backup create prod-db

✘ [ERROR] A request to the Cloudflare API (/accounts/some-id/d1/database/some-id/backup) failed.

  The request is malformed: Experimental DBs do not support backups yet:
  some-id (v3) [code: 7400]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

✘ [ERROR] A request to the Cloudflare API (/accounts/some-id/d1/database/some-id/backup) failed.

  The request is malformed: Experimental DBs do not support backups yet:
  some-id (v3) [code: 7400]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

I'm using wrangler 3.11.0wrangler 3.11.0. Is this error expected? Are backups not yet supported?

James•10/6/23, 5:00 PM

This is expected, yes. Time Travel is the new approach here: https://developers.cloudflare.com/d1/learning/time-travel/

Time Travel and backups · Cloudflare D1 docs

Time Travel is D1’s approach to backups and point-in-time-recovery, and allows you to restore a database to any minute within the last 30 days.

Bbenthecarman Saw you guys made d1 open beta, did you fix the issue where it'll go down for 20...

Matt Silverlock•10/6/23, 5:08 PM

We haven’t seen any reoccurring instances of this - as we said at the time, this was a very rare (but frustrating) bug.

Bbkyerv hello everyone, has anyone had experience setting up drizzle orm with d1? what d...

Matt Silverlock•10/6/23, 5:08 PM

https://github.com/drizzle-team/drizzle-orm/blob/main/examples/cloudflare-d1/README.md

GitHub

drizzle-orm/examples/cloudflare-d1/README.md at main · drizzle-team...

TypeScript ORM that feels like writing SQL. Contribute to drizzle-team/drizzle-orm development by creating an account on GitHub.

JJames This is expected, yes. Time Travel is the new approach here: https://developers....

EvanPiro•10/6/23, 5:10 PM

I'm trying to pull a backup from production and load it on local. is this possible with time-travel?

James•10/6/23, 5:11 PM

Not to my knowledge. You might want to try a third-party community solution: https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

JJames Not to my knowledge. You might want to try a third-party community solution: htt...

EvanPiro•10/6/23, 5:18 PM

Is there a way to pull backups from the dashboard?

James•10/6/23, 5:18 PM

No, Time Travel is the only thing available to new databases at this time in that realm of thought. The community solution I linked above is probably what you're looking for if you need a local copy for now.

benthecarmanOP•10/6/23, 5:27 PM

It happened to us once a week lol

WillTolmie•10/6/23, 5:50 PM

d1 replication

JJames No, Time Travel is the only thing available to new databases at this time in tha...

EvanPiro•10/6/23, 6:11 PM

Thank you, I created a worker that uses the package and am now getting backups to R2 by calling the worker from remote (with auth). Do you know of any commonly used techniques for extending the wrangler CLI so we can add features like this to the runtime and invoke it from the shell/CI?

James•10/6/23, 6:13 PM

I do not unfortunately. I'd probably just create a

scripts

scripts

folder for myself/team to use these kinds of things, personally. You could probably write something that wraps wrangler, but I don't think the effort would be worth it, personally

JJames I do not unfortunately. I'd probably just create a `scripts` folder for myself/t...

EvanPiro•10/6/23, 6:19 PM

How do you go about invoking the worker from local and having it run the backup on remote resources? Also is it possible to pull R2 data from remote to local from wrangler?

James•10/6/23, 6:20 PM

I don't have this exact use-case, but perhaps you could use

--remote

--remote

? Or put the backup script behind some kind of auth and have a local script hit that?

James•10/6/23, 6:21 PM

As for R2 data, I don't think there's an easy way to do that with wrangler or similar, no. You would have to use something to download the contents, perhaps via rclonerclone or similar, and then reconstruct the metadata needed locally. You could try asking in #r2 if anyone has any specific recommendations here.

JJames I don't have this exact use-case, but perhaps you could use `--remote`? Or put t...

EvanPiro•10/6/23, 6:34 PM

The use-case is replicating some pretty complex state on prod for development on local.

EvanPiro•10/6/23, 6:36 PM

I'm think because I now have a worker that performs the back up, I could also have one that pulls the data and have a script on local which calls both and effectively runs a sync.

Aa3x Why doesn't authjs with the drizzel adapter work with Cloudflare D1? I am gettin...

Matt Silverlock•10/6/23, 11:32 PM

I’d open an issue on the Auth.js repo with your code, a reproduction and package versions.

malone•10/7/23, 3:20 AM

hey, i'm a bit confused on storage limits - i'm on workers paid

Limits page says 2GB, full stop
https://developers.cloudflare.com/d1/platform/limits/

Pricing says 5GB included, then $0.75/gb/mo
https://developers.cloudflare.com/d1/platform/pricing/

which is it?

Mmalone hey, i'm a bit confused on storage limits - i'm on workers paid Limits page say...

Cyb3r-Jak3•10/7/23, 3:26 AM

The 2 GB is per database and the 5 GB is the free quota. So if you have 2x2GB databases then it meets the free plan but you can't have a 3 GB database

CCyb3r-Jak3 The 2 GB is per database and the 5 GB is the free quota. So if you have 2x2GB da...

malone•10/7/23, 3:30 AM

aha gotcha! so with one database i can’t go over 2gb even if i pay?

Mmalone aha gotcha! so with one database i can’t go over 2gb even if i pay?

Cyb3r-Jak3•10/7/23, 3:37 AM

For now yes. Limit can increase during the beta but can't say for certain as I'm not an employee.

malone•10/7/23, 3:38 AM

dang i’m already at like 100mb in testflight so im worried about 2gb limit. i think i might migrate before launching publicly to save having to migrate later

malone•10/7/23, 3:38 AM

thanks for clarifying!

Adi//•10/7/23, 6:55 AM

I am getting this error while using a complex SQL statement in wrangler D1

"D1_ERROR: A prepared SQL statement must contain only one statement."
Solved: The ;; at the end of the SQL statement was breaking it

My statement looks like:

let sql_query = "SELECT *
FROM table_name
WHERE id = ?
  AND timestamp > (
    SELECT MAX(timestamp)
    FROM table_name
  );"

let sql_query = "SELECT *
FROM table_name
WHERE id = ?
  AND timestamp > (
    SELECT MAX(timestamp)
    FROM table_name
  );"

Can I not use combined statements?

Adi//•10/7/23, 7:20 AM

gaah it was just the ";" at the end of the string lol

Samantas5855•10/7/23, 10:19 AM

We created our first D1 database and while we haven't inserted any data or used it yet, it has counted 92 reads

Samantas5855•10/7/23, 10:19 AM

how does that work?

SSamantas5855 We created our first D1 database and while we haven't inserted any data or used ...

Isaac McFadyen•10/7/23, 11:08 AM

If you've used the console at all then that will happen.

IIsaac McFadyen If you've used the console at all then that will happen.

Samantas5855•10/7/23, 11:22 AM

We've only created an empty table through the dashboard, no other action was performed

SSamantas5855 We've only created an empty table through the dashboard, no other action was per...

Isaac McFadyen•10/7/23, 11:26 AM

If you even used the console then you'll see rows read, because when the console loads it introspects your database.

IIsaac McFadyen If you even used the console then you'll see rows read, because when the console...

Samantas5855•10/7/23, 11:29 AM

Thank you, do you happen to know whether row deletes count as writes or not?

SSamantas5855 Thank you, do you happen to know whether row deletes count as writes or not?

Matt Silverlock•10/7/23, 11:49 AM

Yes, deletes are writes.

SSamantas5855 Thank you, do you happen to know whether row deletes count as writes or not?

Matt Silverlock•10/7/23, 11:51 AM

Also keep in mind that the first 25 billion reads are included in the basic paid plan, so you could refresh the dash 271 million times a month and be OK

MMatt Silverlock Yes, deletes are writes.

Samantas5855•10/7/23, 12:06 PM

Thank you very much. One last question: Since we are migrating our python scripts from mongodb to D1 is there any way to mass insert rows from an external script using the api, outside of workers?

SSamantas5855 Thank you very much. One last question: Since we are migrating our python script...

Matt Silverlock•10/7/23, 12:08 PM

Yes, you can query from the REST API: https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database#Request

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

SSamantas5855 Thank you very much. One last question: Since we are migrating our python script...

Matt Silverlock•10/7/23, 12:09 PM

e.g. via a Python or Go client. Read from Mongo, slice up the rows, insert.

MMatt Silverlock e.g. via a Python or Go client. Read from Mongo, slice up the rows, insert.

Samantas5855•10/7/23, 12:13 PM

Clarification, we aren't transferring any data from mongodb, just migrating the scripts. Is there a way to insert many rows with one API request?

Paul•10/7/23, 1:12 PM

I’m thinking about swapping my Postgres out for D1. Can I still perform large queries across all my users? Any good UI like Dbeaver?

Nikil•10/7/23, 3:09 PM

Why does D1 use SQLite vs Postgres or MySQL? Unless I misunderstood the docs

Nikil•10/7/23, 3:16 PM

Is 2GB per table very much? I've never really built anything at a scale of more than 1k users so I have no idea.

My best comparision metric right now is a CSV that I have with 82 columns and 101k rows listing every single K12 school in america. That CSV is 46MB, and the Google Sheets version is apparently 16MB

NNikil Why does D1 use SQLite vs Postgres or MySQL? Unless I misunderstood the docs

Hello, I’m Allie!•10/7/23, 3:25 PM

SQLite is a lot easier to embed into a platform(and much lighter) than other DB Systems

HHello, I’m Allie!SQLite is a lot easier to embed into a platform(and much lighter) than other DB ...

Nikil•10/7/23, 3:26 PM

easier in what way?

NNikil Is 2GB per table very much? I've never really built anything at a scale of more ...

forslunds•10/7/23, 3:29 PM

All depends how much data you want to save. I have 8 users and around 3GB in the database. It will be higher limit in the future but also already supports alot of DBs so you could setup one per user (soon)

Fforslunds All depends how much data you want to save. I have 8 users and around 3GB in the...

Nikil•10/7/23, 3:32 PM

One per user - that sounds like a crazy pain to keep all of those schemas in sync?

Nikil•10/7/23, 3:35 PM

And it also sounds incredibly un-performant for specific types of queries at least

Nikil•10/7/23, 3:39 PM

Hmm I guess that's a relatively smart way to shard things, especially if a user is almost always in a defined location.

E.g. if I was creating software to manage parking garages or movie theaters, it could make sense to have 1DB per theater, and 1DB for the "central office" objects that might need to be referenced for all of the DBs

But it still sounds crazy and seems like it might increase the total # of writes?

forslunds•10/7/23, 3:41 PM

Or you could just use the DB you think works best for your solution and slap hyperdrive on it?

Saw you guys made d1 open beta, did you fix the issue where it'll go down for 20+ hours?

Similar Threads

Similar Threads

Similar Threads