I don't have this exact use-case, but perhaps you could use `--remote`? Or put the backup script beh

I don't have this exact use-case, but perhaps you could use

--remote

--remote

? Or put the backup script behind some kind of auth and have a local script hit that?

JamesOP•10/6/23, 6:21 PM

As for R2 data, I don't think there's an easy way to do that with wrangler or similar, no. You would have to use something to download the contents, perhaps via

rclone

rclone

or similar, and then reconstruct the metadata needed locally. You could try asking in #r2 if anyone has any specific recommendations here.

JJames I don't have this exact use-case, but perhaps you could use `--remote`? Or put t...

EvanPiro•10/6/23, 6:34 PM

The use-case is replicating some pretty complex state on prod for development on local.

EvanPiro•10/6/23, 6:36 PM

I'm think because I now have a worker that performs the back up, I could also have one that pulls the data and have a script on local which calls both and effectively runs a sync.

Aa3x Why doesn't authjs with the drizzel adapter work with Cloudflare D1? I am gettin...

Matt Silverlock•10/6/23, 11:32 PM

I’d open an issue on the Auth.js repo with your code, a reproduction and package versions.

malone•10/7/23, 3:20 AM

hey, i'm a bit confused on storage limits - i'm on workers paid

Limits page says 2GB, full stop
https://developers.cloudflare.com/d1/platform/limits/

Pricing says 5GB included, then $0.75/gb/mo
https://developers.cloudflare.com/d1/platform/pricing/

which is it?

Mmalone hey, i'm a bit confused on storage limits - i'm on workers paid Limits page say...

Cyb3r-Jak3•10/7/23, 3:26 AM

The 2 GB is per database and the 5 GB is the free quota. So if you have 2x2GB databases then it meets the free plan but you can't have a 3 GB database

CCyb3r-Jak3 The 2 GB is per database and the 5 GB is the free quota. So if you have 2x2GB da...

malone•10/7/23, 3:30 AM

aha gotcha! so with one database i can’t go over 2gb even if i pay?

Mmalone aha gotcha! so with one database i can’t go over 2gb even if i pay?

Cyb3r-Jak3•10/7/23, 3:37 AM

For now yes. Limit can increase during the beta but can't say for certain as I'm not an employee.

malone•10/7/23, 3:38 AM

dang i’m already at like 100mb in testflight so im worried about 2gb limit. i think i might migrate before launching publicly to save having to migrate later

malone•10/7/23, 3:38 AM

thanks for clarifying!

Adi//•10/7/23, 6:55 AM

I am getting this error while using a complex SQL statement in wrangler D1

"D1_ERROR: A prepared SQL statement must contain only one statement."
Solved: The ;
;
at the end of the SQL statement was breaking it

My statement looks like:

let sql_query = "SELECT *
FROM table_name
WHERE id = ?
  AND timestamp > (
    SELECT MAX(timestamp)
    FROM table_name
  );"

let sql_query = "SELECT *
FROM table_name
WHERE id = ?
  AND timestamp > (
    SELECT MAX(timestamp)
    FROM table_name
  );"

Can I not use combined statements?

Adi//•10/7/23, 7:20 AM

gaah it was just the ";" at the end of the string lol

Samantas5855•10/7/23, 10:19 AM

We created our first D1 database and while we haven't inserted any data or used it yet, it has counted 92 reads

Samantas5855•10/7/23, 10:19 AM

how does that work?

SSamantas5855 We created our first D1 database and while we haven't inserted any data or used ...

Isaac McFadyen•10/7/23, 11:08 AM

If you've used the console at all then that will happen.

IIsaac McFadyen If you've used the console at all then that will happen.

Samantas5855•10/7/23, 11:22 AM

We've only created an empty table through the dashboard, no other action was performed

SSamantas5855 We've only created an empty table through the dashboard, no other action was per...

Isaac McFadyen•10/7/23, 11:26 AM

If you even used the console then you'll see rows read, because when the console loads it introspects your database.

IIsaac McFadyen If you even used the console then you'll see rows read, because when the console...

Samantas5855•10/7/23, 11:29 AM

Thank you, do you happen to know whether row deletes count as writes or not?

SSamantas5855 Thank you, do you happen to know whether row deletes count as writes or not?

Matt Silverlock•10/7/23, 11:49 AM

Yes, deletes are writes.

SSamantas5855 Thank you, do you happen to know whether row deletes count as writes or not?

Matt Silverlock•10/7/23, 11:51 AM

Also keep in mind that the first 25 billion reads are included in the basic paid plan, so you could refresh the dash 271 million times a month and be OK

MMatt Silverlock Yes, deletes are writes.

Samantas5855•10/7/23, 12:06 PM

Thank you very much. One last question: Since we are migrating our python scripts from mongodb to D1 is there any way to mass insert rows from an external script using the api, outside of workers?

SSamantas5855 Thank you very much. One last question: Since we are migrating our python script...

Matt Silverlock•10/7/23, 12:08 PM

Yes, you can query from the REST API: https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database#Request

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

SSamantas5855 Thank you very much. One last question: Since we are migrating our python script...

Matt Silverlock•10/7/23, 12:09 PM

e.g. via a Python or Go client. Read from Mongo, slice up the rows, insert.

MMatt Silverlock e.g. via a Python or Go client. Read from Mongo, slice up the rows, insert.

Samantas5855•10/7/23, 12:13 PM

Clarification, we aren't transferring any data from mongodb, just migrating the scripts. Is there a way to insert many rows with one API request?

Paul•10/7/23, 1:12 PM

I’m thinking about swapping my Postgres out for D1. Can I still perform large queries across all my users? Any good UI like Dbeaver?

Nikil•10/7/23, 3:09 PM

Why does D1 use SQLite vs Postgres or MySQL? Unless I misunderstood the docs

Nikil•10/7/23, 3:16 PM

Is 2GB per table very much? I've never really built anything at a scale of more than 1k users so I have no idea.

My best comparision metric right now is a CSV that I have with 82 columns and 101k rows listing every single K12 school in america. That CSV is 46MB, and the Google Sheets version is apparently 16MB

NNikil Why does D1 use SQLite vs Postgres or MySQL? Unless I misunderstood the docs

Hello, I’m Allie!•10/7/23, 3:25 PM

SQLite is a lot easier to embed into a platform(and much lighter) than other DB Systems

HHello, I’m Allie!SQLite is a lot easier to embed into a platform(and much lighter) than other DB ...

Nikil•10/7/23, 3:26 PM

easier in what way?

NNikil Is 2GB per table very much? I've never really built anything at a scale of more ...

forslunds•10/7/23, 3:29 PM

All depends how much data you want to save. I have 8 users and around 3GB in the database. It will be higher limit in the future but also already supports alot of DBs so you could setup one per user (soon)

Fforslunds All depends how much data you want to save. I have 8 users and around 3GB in the...

Nikil•10/7/23, 3:32 PM

One per user - that sounds like a crazy pain to keep all of those schemas in sync?

Nikil•10/7/23, 3:35 PM

And it also sounds incredibly un-performant for specific types of queries at least

Nikil•10/7/23, 3:39 PM

Hmm I guess that's a relatively smart way to shard things, especially if a user is almost always in a defined location.

E.g. if I was creating software to manage parking garages or movie theaters, it could make sense to have 1DB per theater, and 1DB for the "central office" objects that might need to be referenced for all of the DBs

But it still sounds crazy and seems like it might increase the total # of writes?

forslunds•10/7/23, 3:41 PM

Or you could just use the DB you think works best for your solution and slap hyperdrive on it?

forslunds•10/7/23, 3:44 PM

I’m using D1 bc it’s cheap when needing alot of small dbs/projects , and fast everywhere!

Fforslunds Or you could just use the DB you think works best for your solution and slap hyp...

Nikil•10/7/23, 3:53 PM

If I knew what I was doing, I would agree with this sentiment. But I'm a novice at building product-caliber stuff, so I don't know where to begin other than I want it to be simple to deploy and not eat more than like $10k/yr at scale with a million users in the year.

At smaller scales, I want my whole app to cost < $1/user/mo

Basically, I'm building a course app. We teach high schoolers how to build real products such as Wildfire Simulator in Unity or a Webapp. (I already sell my product to school districts, but am trying to build some in-house custom features and better reporting compared to the third-party tools we're currently using, and then also enable some AI tutoring features in the future).

We host videos, interactive exercises, and quizzes basically. It's kind of like a Masterclass + Brilliant.org

We also have a couple of interesting real-time features such as an office hours queue (so as students are going through one of our projects they can ask a question, and then we group similar questions together and answer the group on a live video call).

Would love to design the system right such that we don't have to completely redo stuff in the near future.

NNikil If I knew what I was doing, I would agree with this sentiment. But I'm a novice ...

malone•10/7/23, 3:59 PM

estimate how big your database needs to be (define your schema and ask chatgpt how large the db would be with x rows in each table) if you’re comfortably under 2gb go with d1

Mmalone estimate how big your database needs to be (define your schema and ask chatgpt h...

Nikil•10/7/23, 4:13 PM

you think ChatGPT will really give a reasonable answer for this?

Mmalone estimate how big your database needs to be (define your schema and ask chatgpt h...

Max (@rozenmd)•10/7/23, 4:15 PM

don't rely on LLMs for maths

malone•10/7/23, 4:20 PM

haha a rough estimate

malone•10/7/23, 4:20 PM

and if it’s anywhere near it you’re probably in trouble

malone•10/7/23, 4:20 PM

feels fine for this task. wouldn’t expect it to be exactly right and you shouldn’t expect your estimate of rows in your db to be exactly right

MMax (@rozenmd)don't rely on LLMs for maths

Nikil•10/7/23, 4:21 PM

that's what I figured

Matt Silverlock•10/7/23, 4:22 PM

Common for a “user” row to be 500-1000 bytes. So 2GB (which is the current, beta limitation) gets you to 2 - 4 million users.

Matt Silverlock•10/7/23, 4:22 PM

(Just for that database, before any sharding, or increased per-DB sizing)

MMatt Silverlock (Just for that database, before any sharding, or increased per-DB sizing)

Nikil•10/7/23, 4:27 PM

ok so if I want to track users' progress through sections of a course (and let's say a course has 50 sections)
there would be ~50 rows per user in the user progress table.

Does that mean I'm actually limited to something like 2GB * (1 progress row / 1MB) * (1 user / 50 progress rows) = 2 million / 50 = 40,000 users?

MMatt Silverlock Common for a “user” row to be 500-1000 bytes. So 2GB (which is the current, _bet...

Nikil•10/7/23, 4:27 PM

that's v helpful btw

NNikil ok so if I want to track users' progress through sections of a course (and let's...

itsmatteomanf•10/7/23, 4:29 PM

Wait, I doubt you'd need that much data to store the progress...

I don't have this exact use-case, but perhaps you could use `--remote`? Or put the backup script beh

Similar Threads