Cloudflare Developers•2y ago

unlikely for now

Max (@rozenmd)OP•4/26/24, 7:37 PM

easier to offer millions of dbs than one big one

MMax (@rozenmd)unlikely for now

wong26•4/26/24, 7:40 PM

what if my business's data grow and require more than 10GB size of database? Does that mean I need to export the D1 and find another alternative to host it?

Wwong26 what if my business's data grow and require more than 10GB size of database? Doe...

IEatBeans•4/26/24, 8:16 PM

Looks like you can request to increase limits here:

https://forms.gle/ukpeZVLWLnKeixDu7

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

IEatBeans•4/26/24, 8:16 PM

That form is found at the bottom of this page btw: https://developers.cloudflare.com/d1/platform/limits/

Cloudflare Docs

Limits · Cloudflare D1 docs

If you would like to explore other storage solutions for your application, Cloudflare also offers Workers KV, Durable Objects, and R2.

IIEatBeans Looks like you can request to increase limits here: https://forms.gle/ukpeZVLWL...

Isaac McFadyen•4/26/24, 8:36 PM

Just an FYI that @Max (@rozenmd) is an engineer on the D1 team and if he says it's not likely to be increased in the future then filling out that form won't get you much considering it eventually ends up with his team anyway

Wwong26 what if my business's data grow and require more than 10GB size of database? Doe...

Chaika•4/26/24, 8:37 PM

D1 wants you to spread your load and size throughput multiple D1 DBs

Chaika•4/26/24, 8:37 PM

currently a single D1 DB is a single single-threaded Durable Object as well, so limited in the querys per second aspect as well

Chaika•4/26/24, 8:38 PM

Although it is worth noting D1 currently lacks the dynamic bindings to make such an approach workable using Bindings, would have to use the API (Which goes back to core from all locations/is slow), or bind each separate DB manually

andrewitsover•4/27/24, 8:27 AM

I am basically going to assume then that the prepared statements are only for the parameter binding and don't have any performance usage on d1.

andrewitsover•4/27/24, 8:27 AM

(outside of multiple inserts or something)

andrewitsover•4/27/24, 9:14 AM

because you aren't supposed to store the statement object outside of the request handler..... unless d1 does some pattern matching that is different to sqlite in general

andrewitsover•4/27/24, 9:14 AM

but isn't specified anywhere

andrewitsover•4/27/24, 9:14 AM

and doesnt make sense... because you have to call .prepare each request

BBeny I think andrew is confused because D1 has a different API for it.

andrewitsover•4/27/24, 9:20 AM

errr... you can do the same with d1, but it must only be per request, not between requests. The documentation is here: https://developers.cloudflare.com/d1/build-with-d1/d1-client-api/#reuse-prepared-statements nobody seems to know anything about this topic, including the employee here

Cloudflare Docs

Query D1 · Cloudflare D1 docs

D1 is compatible with most SQLite’s SQL convention since it leverages SQLite’s query engine. D1 client API allows you to interact with a D1 database …

Nick•4/27/24, 10:07 AM

I’m looking to build something with D1 but I’ll almost certainly exceed the 10GB max database size. Is that a hard limit, or can it be increased? If it’s a hard limit, what’s the best solution? Consistent hashing based on key and have N databases?

Nick•4/27/24, 10:08 AM

I would have a database per customer but creating a D1 on the fly seems awful as does applying migrations to all of them

NNick I would have a database per customer but creating a D1 on the fly seems awful as...

bret_pat•4/27/24, 11:16 AM

From my understanding, D1 has a very narrow scope where it's beneificial and the drawbacks that you mentioned are not minor.

Nick•4/27/24, 11:41 AM

Hmm, my use case is I need to check metadata on every request, and there will be 10s-100s of millions of metadata rows. Need it to be fast and cheap. KV better suited I guess?

Nick•4/27/24, 11:42 AM

Having a SQL like Interface would be ideal though

Aandrewitsover errr... you can do the same with d1, but it must only be per request, not betwee...

kian•4/27/24, 12:18 PM

It's been explained several times correctly. The object you get back from env.DB.prepare(...)env.DB.prepare(...) is just an object, it has almost nothing to do with SQLite. It is sent to your database, which makes a prepared statement with the SQL text and binds it with your parameters.

By re-using the same SQL text, you will re-use the prepared statement. (when using prepareprepare both times, of course)

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:36 PM

From what you've said, you haven't excluded using D1 yet.

Things like knowing how performant it needs to be (remember, it's just a single DO doing all the work), and space limitations. Also how big per row you're expecting

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:36 PM

Row limits are 1MB, and then of course, the 10GB limit per db

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:37 PM

100s of millions of rows probably means you need multiple D1 and so D1 is a lot harder to use then

kamama•4/28/24, 11:20 AM

Sharding is your best friend ever

Rubi•4/29/24, 12:27 AM

I'm doing D1 load test 1 INSERT/hit

Ccopper super nice, so d1 maxes out at -300 inserts rpm?!

Rubi•4/29/24, 5:40 AM

not really a maxes out, it can go more depends on the Locust setting

also I'm doing the benchmark from my home PC to Cloudflare Pages with D1 binding, GET / without client payload, so my connection quality take a part there

Ccopper ah ok but still nice

Rubi•4/29/24, 5:41 AM

absolutely

Ccopper wondeirng for weeks if sqlite (d1 or turso) is enough re insert perf or i just s...

Rubi•4/29/24, 5:43 AM

this is exactly what I wondering about in #kv , and I decide to stay with D1 for my startup

Rubi•4/29/24, 5:47 AM

I just can't handle the stress managing VPS's, or even, DB on VPS

Ccopper what's your guess and could you just shotgun the requests from a vps connected t...

Rubi•4/29/24, 5:48 AM

I cant promise, but this is interesting to try

Rubi•4/29/24, 5:51 AM