Cloudflare Developers•2y ago

There are ways, but there are limits... you can statically bind them, but it doesn't scale that well

There are ways, but there are limits... you can statically bind them, but it doesn't scale that well now, unofrtunately

Befus•2/19/24, 10:43 PM

Hmm you will probably hear from me again, just figuring out how I approach this

BBefus Hmm you will probably hear from me again, just figuring out how I approach this

itsmatteomanfOP•2/19/24, 10:48 PM

Do also consider you can use other products at the same time, you don't have to store blobs or JSON in the db if you don't need it in a query. KV, R2, etc. exist.

Iitsmatteomanf Do also consider you can use other products at the same time, you don't have to ...

Befus•2/19/24, 10:50 PM

Well the situation is a bit like this, we were currently using KV but we are moving towards a platform meaning if we fetch for example users we would get everyones users, making it slower, etc. Thats why I thought a SQL type database would help. Not sure though

BBefus Well the situation is a bit like this, we were currently using KV but we are mov...

itsmatteomanfOP•2/19/24, 10:51 PM

Oh yeah, it definitely would. But not all user data needs to live on the db

itsmatteomanfOP•2/19/24, 10:51 PM

Also, D1 isn't yet GA, while it's supposedly close-ish, it's still not fully production ready, technically.

Befus•2/19/24, 10:52 PM

Yea I'm aware but willing to take/accept that risk

Iitsmatteomanf Oh yeah, it definitely would. But not all user data needs to live on the db 🙂

Befus•2/19/24, 10:54 PM

What things would you not put on the db but on other service for example, i'm aware of objects/files I was planning on using cloudflare images/R2 but maybe you have some suggestions?

BBefus What things would you not put on the db but on other service for example, i'm aw...

itsmatteomanfOP•2/19/24, 10:56 PM

That really depends on the data you have, you need to look at the data you have and the requirements for your application

Iitsmatteomanf That really depends on the data you have, you need to look at the data you have ...

Befus•2/19/24, 10:57 PM

Yea true, thanks man!

Iitsmatteomanf There are ways, but there are limits... you can statically bind them, but it doe...

Befus•2/20/24, 8:40 AM

Do you have some docs or anything that explain the possible ways or solutions to dynamically allocate?

BBefus Do you have some docs or anything that explain the possible ways or solutions to...

itsmatteomanfOP•2/20/24, 8:47 AM

There are no ways to dynamically allocate at the moment

Iitsmatteomanf There are no ways to dynamically allocate at the moment

Befus•2/20/24, 8:52 AM

Ah you mentioned that there were ways I thought lmao

BBefus Ah you mentioned that there were ways I thought lmao

itsmatteomanfOP•2/20/24, 8:54 AM

No, there are ways around the limitation

Befus•2/20/24, 8:54 AM

Ohhhh lol

itsmatteomanfOP•2/20/24, 8:57 AM

I do really hope they will fix the limitation at the root, though, it's the biggest D1 issue at the moment

itsmatteomanfOP•2/20/24, 8:57 AM

Secondarily I'd hope for read replicas, but tbh it's secondary.

Befus•2/20/24, 9:03 AM

Im thinking of a way I could somehow dynamicly allocate maybe by using KV or smth but don't know if its possible in any way

BBefus Im thinking of a way I could somehow dynamicly allocate maybe by using KV or smt...

Hello, I’m Allie!•2/20/24, 9:04 AM

No, unless you are willing to go via the HTTP API, which is painfully slow

HHello, I’m Allie!No, unless you are willing to go via the HTTP API, which is *painfully* slow

Befus•2/20/24, 9:49 AM

Just thinking out loud here but if I were to create the database manually before hand, when in my case a new organisation register check for the first empty database and assign that way. You reckon that would be possible?

BBefus Just thinking out loud here but if I were to create the database manually before...

Hello, I’m Allie!•2/20/24, 9:57 AM

Maybe, but that would only work for the first N databases

bun•2/20/24, 10:13 AM

are there no tools compatible with d1 ? datagrip, tableplus, or anything?

bun•2/20/24, 10:15 AM

does worker use TCP for D1 or HTTP? like does it count as an outgoing request?

Original message was deleted

bun•2/20/24, 10:18 AM

holy f

Júlio Mühlbauer•2/20/24, 10:20 AM

Does D1 have this types of limitations because of SQLite? https://twitter.com/heyandras/status/1758940021640274148?t=1C0_ZdZ4JytNJItZrMJcwg&s=19

Andras Bacsai (@heyandras) on X

The only case when Sqlite is not suitable for your project, is when more than one server/instance of your app needs to access the database.

Your app is probably not in this category.

Twitter

•

2/17/24, 7:42 PM

Bbun does worker use TCP for D1 or HTTP? like does it count as an outgoing request?

Hello, I’m Allie!•2/20/24, 10:21 AM

It uses something internally, which afaik isn't quite TCP or HTTP

bun•2/20/24, 10:25 AM

so queries dont count as outgoign requests?

JJúlio Mühlbauer Does D1 have this types of limitations because of SQLite? https://twitter.com/he...

Hello, I’m Allie!•2/20/24, 10:25 AM

Not quite, since the D1 Layer that is on top of SQLite handles simultaneous connections. It can't actually process multiple requests simultaneously, but it can store requests in a queue. I would note too that while you can use D1 outside of Cloudflare, it isn't meant to be used that way, and thus it is a bit unergonomic

Bbun so queries dont count as outgoign requests?

Hello, I’m Allie!•2/20/24, 10:26 AM

They do. No matter the underlying protocol, an outgoing I/O op is counted as a "request"(though opening a TCP socket is a single connection, even if you do multiple things with it)

Hello, I’m Allie!•2/20/24, 10:26 AM

TCP is the only special snowflake, at least at the moment

HHello, I’m Allie!Not quite, since the D1 Layer that is on top of SQLite handles simultaneous conn...

Júlio Mühlbauer•2/20/24, 10:28 AM

Is there some limitations on scalability compared to a postgres database? It would be cool to see this type of thing on the docs. Thanks!

bun•2/20/24, 10:28 AM

are there any standardized approaches on sharding the database currently? it sucks a lot that databases are 2 gb only....

JJúlio Mühlbauer Is there some limitations on scalability compared to a postgres database? It wou...

Hello, I’m Allie!•2/20/24, 10:28 AM

There is https://developers.cloudflare.com/d1/platform/limits/, if that is what you mean

Bbun are there any standardized approaches on sharding the database currently? it suc...

Hello, I’m Allie!•2/20/24, 10:29 AM

No, though generally the optimal sharding strategy is dependent on the shape of the data you are storing

bun•2/20/24, 10:29 AM

like do i have to do 100 bindings in my wrangler.toml

Bbun like do i have to do 100 bindings in my wrangler.toml

Hello, I’m Allie!•2/20/24, 10:29 AM

That is a way. You can also offload large values to a separate store, if you don't need to query on them directly

HHello, I’m Allie!There is https://developers.cloudflare.com/d1/platform/limits/, if that is what ...

Júlio Mühlbauer•2/20/24, 10:30 AM

As I undertood, this is mostly limits impose by cloudflare that can be increased by a request or paying. But not about the technology the database uses, right?

JJúlio Mühlbauer As I undertood, this is mostly limits impose by cloudflare that can be increased...

Hello, I’m Allie!•2/20/24, 10:32 AM

Not necessarily. While some of the limits are applied by Cloudflare directly, others are in place because the database may not be stable above those limits. You might be able to push it farther, but CF can't guarantee that it will function correctly, so you can't raise the limit that far

HHello, I’m Allie!Not necessarily. While some of the limits are applied by Cloudflare directly, ot...

Júlio Mühlbauer•2/20/24, 10:32 AM

Understand. Thanks!

bun•2/20/24, 10:32 AM

are there no estimates on increasing the storage of the database?

Bbun are there no estimates on increasing the storage of the database?

Hello, I’m Allie!•2/20/24, 10:33 AM

Estimates of Pricing? Timeline? Other?

bun•2/20/24, 10:33 AM

timeline

Bbun timeline

Hello, I’m Allie!•2/20/24, 10:33 AM

:soontm:

Júlio Mühlbauer•2/20/24, 10:33 AM

Would love to see a postgres or something that scales made by cloudflare btw

Original message was deleted

Befus•2/20/24, 10:59 AM

I will have a look at this, thanks a lot!

Original message was deleted

Befus•2/20/24, 11:03 AM

That will be a solution to the waiting for deployment?

Befus•2/20/24, 11:04 AM

Awesome!

Original message was deleted

Max (@rozenmd)•2/20/24, 12:09 PM

here's how we load the entire northwind db (via the REST API/wrangler): https://raw.githubusercontent.com/cloudflare/d1-northwind/main/db/data-big.sql

Vignesh•2/20/24, 12:12 PM

Any ETA on official dynamic binding?

Original message was deleted

Befus•2/20/24, 12:25 PM

Gonna be trying this out these next few days first gotta setup next auth with d1

Original message was deleted

itsmatteomanfOP•2/20/24, 1:07 PM

Oh yeah, if you do a batch with reused statements especially, it’s in the 100s of thousands of inserts per call in a lot cases. Performance wise is really good.

There are ways, but there are limits... you can statically bind them, but it doesn't scale that well

Similar Threads

Similar Threads

Similar Threads