D1 wants you to spread your load and size throughput multiple D1 DBs

ChaikaOP•4/26/24, 8:37 PM

currently a single D1 DB is a single single-threaded Durable Object as well, so limited in the querys per second aspect as well

ChaikaOP•4/26/24, 8:38 PM

Although it is worth noting D1 currently lacks the dynamic bindings to make such an approach workable using Bindings, would have to use the API (Which goes back to core from all locations/is slow), or bind each separate DB manually

andrewitsover•4/27/24, 8:27 AM

I am basically going to assume then that the prepared statements are only for the parameter binding and don't have any performance usage on d1.

andrewitsover•4/27/24, 8:27 AM

(outside of multiple inserts or something)

andrewitsover•4/27/24, 9:14 AM

because you aren't supposed to store the statement object outside of the request handler..... unless d1 does some pattern matching that is different to sqlite in general

andrewitsover•4/27/24, 9:14 AM

but isn't specified anywhere

andrewitsover•4/27/24, 9:14 AM

and doesnt make sense... because you have to call .prepare each request

BBeny I think andrew is confused because D1 has a different API for it.

andrewitsover•4/27/24, 9:20 AM

errr... you can do the same with d1, but it must only be per request, not between requests. The documentation is here: https://developers.cloudflare.com/d1/build-with-d1/d1-client-api/#reuse-prepared-statements nobody seems to know anything about this topic, including the employee here

Cloudflare Docs

Query D1 · Cloudflare D1 docs

D1 is compatible with most SQLite’s SQL convention since it leverages SQLite’s query engine. D1 client API allows you to interact with a D1 database …

Nick•4/27/24, 10:07 AM

I’m looking to build something with D1 but I’ll almost certainly exceed the 10GB max database size. Is that a hard limit, or can it be increased? If it’s a hard limit, what’s the best solution? Consistent hashing based on key and have N databases?

Nick•4/27/24, 10:08 AM

I would have a database per customer but creating a D1 on the fly seems awful as does applying migrations to all of them

NNick I would have a database per customer but creating a D1 on the fly seems awful as...

bret_pat•4/27/24, 11:16 AM

From my understanding, D1 has a very narrow scope where it's beneificial and the drawbacks that you mentioned are not minor.

Nick•4/27/24, 11:41 AM

Hmm, my use case is I need to check metadata on every request, and there will be 10s-100s of millions of metadata rows. Need it to be fast and cheap. KV better suited I guess?

Nick•4/27/24, 11:42 AM

Having a SQL like Interface would be ideal though

Aandrewitsover errr... you can do the same with d1, but it must only be per request, not betwee...

kian•4/27/24, 12:18 PM

It's been explained several times correctly. The object you get back from env.DB.prepare(...)env.DB.prepare(...) is just an object, it has almost nothing to do with SQLite. It is sent to your database, which makes a prepared statement with the SQL text and binds it with your parameters.

By re-using the same SQL text, you will re-use the prepared statement. (when using prepareprepare both times, of course)

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:36 PM

From what you've said, you haven't excluded using D1 yet.

Things like knowing how performant it needs to be (remember, it's just a single DO doing all the work), and space limitations. Also how big per row you're expecting

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:36 PM

Row limits are 1MB, and then of course, the 10GB limit per db

NNick Hmm, my use case is I need to check metadata on every request, and there will be...

bret_pat•4/27/24, 12:37 PM

100s of millions of rows probably means you need multiple D1 and so D1 is a lot harder to use then

kamama•4/28/24, 11:20 AM

Sharding is your best friend ever

Rubi•4/29/24, 12:27 AM

I'm doing D1 load test 1 INSERT/hit

Ccopper super nice, so d1 maxes out at -300 inserts rpm?!

Rubi•4/29/24, 5:40 AM

not really a maxes out, it can go more depends on the Locust setting

also I'm doing the benchmark from my home PC to Cloudflare Pages with D1 binding, GET / without client payload, so my connection quality take a part there

Ccopper ah ok but still nice

Rubi•4/29/24, 5:41 AM

absolutely

Ccopper wondeirng for weeks if sqlite (d1 or turso) is enough re insert perf or i just s...

Rubi•4/29/24, 5:43 AM

this is exactly what I wondering about in #kv , and I decide to stay with D1 for my startup

Rubi•4/29/24, 5:47 AM

I just can't handle the stress managing VPS's, or even, DB on VPS

Ccopper what's your guess and could you just shotgun the requests from a vps connected t...

Rubi•4/29/24, 5:48 AM

I cant promise, but this is interesting to try

Rubi•4/29/24, 5:51 AM

yes, I undserstood, I migbt be using locust on vps, but I cant promise this

Rubi•4/29/24, 5:53 AM

nope, it was just a basic .prepare insert

Rubi•4/29/24, 5:53 AM

I havent take a look the tx session feature

Kimitri•4/29/24, 7:06 AM

Hellow, I have a Node.JS Express API runing in a docker in a VPS with a mysql db. I wanna try D1 for the first time, the problem is, to migrate my current API and DB for D1 will I need to rewrite the entire api to workers ?