Cloudflare Developers•16mo ago

there is no way that would be possible, imagine a large restaurant chain, very likely it would need

there is no way that would be possible, imagine a large restaurant chain, very likely it would need a lot of space

�

𝓈𝒽𝓎OP•9/10/24, 3:24 PM

and having multiple under one db that is limited to 10GB doesn't sound like a good idea

�

𝓈𝒽𝓎OP•9/10/24, 3:28 PM

besides, I mainly wanted the database per business to easy separation and being able to rollback for that specific business, also give them all their data if they wanted to export it

�𝓈𝒽𝓎 there is no way that would be possible, imagine a large restaurant chain, very l...

o7•9/10/24, 3:33 PM

thats why I asked how much you were storing per user/row

o7•9/10/24, 3:34 PM

i can personally store over 50 million with the basic amount of data im writing per user

o7•9/10/24, 3:34 PM

ive only ever tested to 500,000 rows and did the math with how much space it used

�

𝓈𝒽𝓎OP•9/10/24, 3:35 PM

are you talking about a single table?

o7•9/10/24, 3:37 PM

yes

�

𝓈𝒽𝓎OP•9/10/24, 3:37 PM

that's not a realistic test

o7•9/10/24, 3:38 PM

it was meant to be a theoretical test

o7•9/10/24, 3:43 PM

using a DB per user would be awkward to manage IMO. At that point I would use KV/DO.

storing all users in a single table/sharding the table makes more sense as you'd be able to query on every row without having to jump between DBs, especially considering the cold start times

�

𝓈𝒽𝓎OP•9/10/24, 3:46 PM

if I couldn't find a good solution for this I might consider switching

John Spurlock•9/10/24, 5:41 PM

curious what you mean by this - do you mean exporting it to a local .db file? what's complicated about the one liner wrangler command (other than .sql -> .db step at the end)

JJohn Spurlock curious what you mean by this - do you mean exporting it to a local .db file? w...

ac•9/10/24, 6:47 PM

Yes, I mean exporting it to a simple .db file. The export from the wrangler command just exports queries, which in my experience are prone to issues if the dataset is large enough. It creates a lot of extra work, where a simple sqlite file would be amazing for dev experience

birdfly•9/11/24, 10:34 AM

Hello ,I was charged 1347.30 USD for D1 - Rows Read ,but the 1,342,795,529,750 mentioned in the invoice doesn't match what I saw in cloudflare dashboard ,where I can get help

James•9/11/24, 12:31 PM

Is this an expected number of errors with this number of requests? https://community.cloudflare.com/t/multiple-errors-from-worker-d1-error-network-connection-lost/710299?u=cherryjimbo

nulled•9/11/24, 4:31 PM

Hi All. Hopefully this is the right place to ask this question:

I'm working on a project where a local python app uses the cloudflare D1 REST API (via cloudflare-python) to write data to D1 as the producer.

I then use cloudflare workers to query D1 to consume this data and display it on a web app.

D1 is amazing! Everything "works" however the D1 REST API via the python app seems quite slow, perhaps hitting some type of REST API rate limit? So my next step was to batch my D1 queries in the python app to reduce round trips, same as I do in my workers. But this does not appear to be supported via the rest API? https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database

The REST API spec includes "Your SQL query. Supports multiple statements, joined by semicolons, which will be executed as a batch."
Great! But how do I pass in multiple params arrays for each different query? I definitely want to be using prepared statements here for injection safety.

I could write a new worker to make a bespoke API that I send my queries and params to, and then batch them via the worker, but I was hoping to avoid another worker to maintain.

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

nulled•9/11/24, 4:38 PM

I found a similar question from the community forums here: https://community.cloudflare.com/t/d1-batch-mode-in-rest-api/697481 but sadly it is unanswered.

Aac Yes, I mean exporting it to a simple .db file. The export from the wrangler comm...

John Spurlock•9/11/24, 4:49 PM

gotcha - I agree. I'm in the middle of tweaking D1 support in my denoflaredenoflare tool, and it wasn't too hard to add a single one-liner subcommand for exporting a D1 database to a local sqlite DB file: denoflare d1 export-db your-database-name --file /path/to/downloaded.dbdenoflare d1 export-db your-database-name --file /path/to/downloaded.db

details here if you want to try it out: https://gist.github.com/johnspurlock-skymethod/2e3f1a1d9762f8034d622a53ca7aa1a2

Gist

Export a D1 database to a local sqlite3 db file

Export a D1 database to a local sqlite3 db file. GitHub Gist: instantly share code, notes, and snippets.

JJohn Spurlock gotcha - I agree. I'm in the middle of tweaking D1 support in my `denoflare` to...

ac•9/11/24, 4:53 PM

I'll check it out, thanks for sharing!

Nnulled Hi All. Hopefully this is the right place to ask this question: I'm working on...

Dean•9/11/24, 11:53 PM

accessing D1 via the CF API is slow because it has to route to a core location. Create an API with a cloudflare worker and bind the database to the worker and it'll be significantly quicker.

Ppetar Hi folks. My team and I are starting to use D1 where we need to have one simple ...

Gerhard•9/12/24, 5:24 AM

Aren’t you more interested in sending sql cmds to update the table? Why the import?

POST
https://api.cloudflare.com/client/v4
/accounts/{account_id}/d1/database/{database_id}/query

�𝓈𝒽𝓎 hello @Max (@rozenmd) I'm currently building my startup which would be a POS saa...

�

𝓈𝒽𝓎OP•9/12/24, 5:45 AM

Anyone from cloudflare team can update me about this please

Mrinank•9/12/24, 5:56 AM

can we store images in d1?

MMrinank can we store images in d1?

�

𝓈𝒽𝓎OP•9/12/24, 6:17 AM

you should probably use r2 or images and store the link in d1, storing directly in any db takes a lot of storage, and in d1 being limited to 10GB doesn't seem like a good idea

�𝓈𝒽𝓎 you should probably use r2 or images and store the link in d1, storing directly ...

Mrinank•9/12/24, 7:04 AM

Yeah, but I only wanted to try it for my side projects

anand248•9/12/24, 9:00 AM

Hey Everyone, is there a plan to support Jurisdiction Restriction on D1 database in the future?

�

𝓈𝒽𝓎OP•9/12/24, 10:24 AM

does cloudflare team help if I ask a question here?

�𝓈𝒽𝓎 does cloudflare team help if I ask a question here?

1984 Ford Laser•9/12/24, 10:48 AM

You're definitely more likely to hear from your fellow community members. Depending on the issue, and the time of day, the right person might see and reply to you

1984 Ford Laser•9/12/24, 10:49 AM

Unrelated, has anyone bulk moved data from mongoDB to D1? Can't seem to find any decent info online

�𝓈𝒽𝓎 does cloudflare team help if I ask a question here?

Isaac McFadyen•9/12/24, 12:12 PM

Employees volunteer their time here, it's not part of their obligations as employees - if you don't hear from them it's probably because they're busy or not available at the moment, or they're just not on Discord

oof2win2•9/12/24, 2:29 PM

hi, i've searched on the internet but there doesn't seem to be a solution to the D1_ERROR: Network connection lost.D1_ERROR: Network connection lost. error. i don't think that this is supposed to happen, can someone please explain it to me?

oof2win2•9/12/24, 2:30 PM

i'm executing ~300 requests in parallel to simply run this (300 requests to the worker, each worker runs the query only once)

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

Ooof2win2 i'm executing ~300 requests in parallel to simply run this (300 requests to the ...

Isaac McFadyen•9/12/24, 3:38 PM

You should probably run in fewer statements, each inserting more than one record. A few requests inserting more than one item will be more efficient than many requests all inserting one item.

oof2win2•9/12/24, 3:44 PM

yeah but i run one insert per http request

oof2win2•9/12/24, 3:44 PM

cant really optimize that better

Ooof2win2 yeah but i run one insert per http request

Isaac McFadyen•9/12/24, 3:47 PM

Oh I see, I thought you were talking about bulk inserting data but you mean that each request actually only generates 1 row to insert...

bovium•9/12/24, 5:03 PM

Hi I would like to know if there is a way to remove data back in time in the D1 time travel backup databases.

In these days of GDPR it is a very relevant feature that you are able to remove data from your backups.

IIsaac McFadyen Oh I see, I thought you were talking about bulk inserting data but you mean that...

oof2win2•9/12/24, 5:09 PM

yeah my bad thats what i mean

ysm•9/13/24, 3:40 AM

How to get

meta

meta

object (duration, row_read, row_written) in D1 + Prisma query?

1984 Ford Laser•9/13/24, 7:54 AM

I have a bunch of data in a mongo install that I want to move to D1. For ease of development and saving my sanity, is there any downside to storing all data in D1 as JSON? and just querying the json as necessary? Would have index columns as independent columnds outside of the JSON column. Mainly wondering about performance issues that may arise