Thanks, any ETA on raising the storage per-database limit? We cannot take the risk to develop on D1

Thanks, any ETA on raising the storage per-database limit?
We cannot take the risk to develop on D1 if the limit is not raised once we go in production (late 2023)

kian•11/8/23, 11:27 AM

I’m not aware of a specific timeline, although Matt may be able to share one. It’s currently a limit that can’t be raised by exception (which is to say it applies to everyone and isn’t a case-by-case basis limit).

Kkian I’m not aware of a specific timeline, although Matt may be able to share one. It...

Lila RestOP•11/8/23, 11:51 AM

Thanks, so I'll wait for Matt's answer

Ravi kiran•11/8/23, 12:19 PM

Hey guys i just started using cloudflare workers for a project and im using cloudflare D1 as the db. My doubt is regarding D1.
i want a field in a table suppose the users table wherein i want the field to be store multiple strings in an array. i.e ["123","456","789"]. i tried it like as shown below

CREATE TABLE IF NOT EXISTS Study(
_id VARCHAR(20) PRIMARY KEY,
services_provided TEXT[]
);
now i want to query the record based on a value in the array i.e services_provided.
Can anyone please give me the query for this? i have tried to use the array field and the json filed but it does not work.

Ravi kiran•11/8/23, 12:26 PM

Can anyone answer this please

kian•11/8/23, 12:39 PM

SQLite doesn't support arrays natively - you probably want to use JSON instead.

kian•11/8/23, 12:39 PM

Have you tried json_eachjson_each? https://stackoverflow.com/a/63653966

Ravi kiran•11/8/23, 12:40 PM

nope. i will try using this and get back.

Ravi kiran•11/8/23, 12:45 PM

Thanks, this works.

LLila Rest Thanks, any ETA on raising the storage per-database limit? We cannot take the ri...

Matt Silverlock•11/8/23, 12:46 PM

What per database size are you looking for?

forslunds•11/8/23, 12:58 PM

Myself would love either A) 5GB (good standard size and room/time to adjust) or B) keep 2gb but make sure its easy to make one-db-per-user-bindings

Rick•11/8/23, 1:19 PM

For something like sessions, should I be using kv or d1? kv sounds like the obvious choice but I wanted to make sure that is much more performant that d1 for simple lookups

RRick For something like sessions, should I be using kv or d1? kv sounds like the obvi...

GuillaumeLakano•11/8/23, 1:21 PM

KV keys need to be loaded a lot of time per minute to be « hot » and fast, so I don't think it's a good idea

RRick For something like sessions, should I be using kv or d1? kv sounds like the obvi...

GuillaumeLakano•11/8/23, 1:22 PM

https://developers.cloudflare.com/workers/learning/storage-options/ for storage comparaison

MMatt Silverlock What _per database_ size are you looking for?

Lila RestOP•11/8/23, 1:23 PM

We are building a SaaS, so ideally the DB should be able to scale infinitely with our userbase and the content they create.
But maybe am I not well understanding how D1 should be properly used? Maybe 1 DB per user would be the good way to do it?

Lila RestOP•11/8/23, 1:25 PM

Anyway, it looks that even 2GB per users could be reached by our oldest and most intensive users.

GGuillaumeLakano KV keys need to be loaded a lot of time per minute to be « hot » and fast, so I ...

Rick•11/8/23, 1:27 PM

Not sure I understood. Since each route request will trigger a session check, wouldn't that mean that each session in my kv will be "hot"?

GGuillaumeLakano KV keys need to be loaded a lot of time per minute to be « hot » and fast, so I ...

Matt Silverlock•11/8/23, 1:28 PM

That’s not true. “A lot of times per minute” is definitely an exaggeration.

MMatt Silverlock That’s not true. “A lot of times per minute” is definitely an exaggeration.

GuillaumeLakano•11/8/23, 1:34 PM

So may be it's changed since february ?

GuillaumeLakano•11/8/23, 1:37 PM

From what I understood of this answer of february, is we need to read at least 2 times in 60 seconds a KV key to be considered as « hot » and be fast. May be it's changed or may be I didn't understood correctly

kian•11/8/23, 1:43 PM

KV is eventually consistent, with a central stores in NA and EU. When you read a value in location A, it will be cached for at least 60 seconds.

This means that if you read it in Colo A, write it to a new value in Colo B, Colo A's reads will still show the old value for up to 60 seconds.

kian•11/8/23, 1:44 PM

Compared to Durable Objects, R2 and D1 where they are strongly consistent (read-after-write) and all reads/writes go to the central stores. R2 and D1 may introduce their own replication down the line, which allows you to have a trade-off between consistency and latency to suit your use-case.

kian•11/8/23, 1:46 PM

The How KV works document still has some references to tiers, similar to https://developers.cloudflare.com/cache/how-to/tiered-cache/, which an experimental version of KV used but isn't currently available.

GGuillaumeLakano From what I understood of this answer of february, is we need to read at least 2...

Matt Silverlock•11/8/23, 1:46 PM

There's no such metric. Cached values can remain in the cache after just one read: we don't make guarantees that they will stay in cache all the time if they are infrequently accessed.

MMatt Silverlock There's no such metric. Cached values can remain in the cache after just one rea...

GuillaumeLakano•11/8/23, 1:46 PM

Ok, good to known, thank you for the precision!

Rick•11/8/23, 1:46 PM

so, is kv still the better choise for storing sessions?

RRick so, is kv still the better choise for storing sessions?

Max (@rozenmd)•11/8/23, 1:50 PM

does your use case tolerate data being out of date for up to 60 seconds?

RRick so, is kv still the better choise for storing sessions?

GuillaumeLakano•11/8/23, 1:56 PM

If the performance is important for you, I suggest you to compare the speed between a session stored in a KV, and stored in D1 ( even if D1 isn't yet replicated )
Also, there is other options off of storage, like JWTs / Biscuit ...

johnswarbrick•11/8/23, 2:39 PM

Are there any plans to add spacial data support to D1 for geo-lookups? Maybe adding an extension like Spatialite (ideally), or enabling R*Tree support?

Jjohnswarbrick Hi 👋 Are there any plans to add spacial data support to D1 for geo-lookups? Ma...

Matt Silverlock•11/8/23, 2:46 PM

More likely R*Tree, but no formal plans yet

LLila Rest We are building a SaaS, so ideally the DB should be able to scale infinitely wit...

Lila RestOP•11/8/23, 2:47 PM

Bumping up this message @Matt

MMatt Silverlock More likely R*Tree, but no formal plans _yet_

johnswarbrick•11/8/23, 3:09 PM

Regarding replication in D1, I understand it's not replicated currently, but I assume the plan is to eventually have read-replicas in each geo to ensure that queries run quickly within that particular location rather than having to go all the way back to the central core?

Jökull Sólberg•11/8/23, 3:39 PM

It's crazy there is no good way to get a backup from D1 anymore. Why not at least support .dump.dump in execute?

JJökull Sólberg It's crazy there is no good way to get a backup from D1 anymore. Why not at leas...

Cyb3r-Jak3•11/8/23, 3:56 PM

I believe it’s due to how the new backend stores data with time travel. There are a couple community projects that offer the ability to dump the DB.

Jökull Sólberg•11/8/23, 3:58 PM

This feels like lock in - I'm getting errors using the community projects, and they also have limitations according to the README's. I hope the blunt feedback is appreciated on some level. I know there are amazing people working on this project and I've been very supportive on Twitter thus far. I'm also a bit frustrated because there's a regression - this used to be very simple and easy.

Jökull Sólberg•11/8/23, 3:59 PM

I'm now doing execute --command select * from ...execute --command select * from ... for each table and writing import scripts. So I'm not blocked per se ... just expected to have some way to dump the db

Cyb3r-Jak3•11/8/23, 4:03 PM

It is planned to have it just no ETA for it

JJökull Sólberg This feels like lock in - I'm getting errors using the community projects, and t...

Matt Silverlock•11/8/23, 4:04 PM

It's planned by GA.

kian•11/8/23, 4:04 PM

.dump.dump is more or less doing the same thing as the exporters. It's purely a SQLite3 shell command, rather than being available in their C API (which embedders like Workers use) so it would need the functionality written up rather than being something they can enable or use from SQLite.

Kkian `.dump` is more or less doing the same thing as the exporters. It's purely a SQL...

Matt Silverlock•11/8/23, 4:04 PM

Additionally, on the "alpha" backend, backups would constantly fail due to the limitations with that backend. We built a new storage engine that is reliable + far, far faster + enables PITR as a priority over exporting backups during the beta.

kian•11/8/23, 4:05 PM

The shell has a lot of niceties ( including being overly permissive on single vs double quotes, :-( ) which embedders have to reproduce

kian•11/8/23, 4:05 PM

Jökull Sólberg•11/8/23, 4:07 PM

Ok thanks for the clarification. Good point about GA. I'm hyped about D1

Kkian The shell has a lot of niceties ( including being overly permissive on single vs...

kian•11/8/23, 4:13 PM

The double-quoted string misfeature is now disabled by default for CLI builds

nevermind, thankfully that's gone in 3.41.0

kian•11/8/23, 4:20 PM

There's a lot of cool functions that'll land if/when D1 moves from 3.40.1 to 3.44.0

kian•11/8/23, 4:20 PM

unhex, base64, timediff, concat, etc

Victor•11/8/23, 5:01 PM

@kian In the sqlite upgrade pr, what is the allowlist that you are mentioned

kian•11/8/23, 5:02 PM

kian•11/8/23, 5:02 PM

Functions are explicitly allowed.

MMax (@rozenmd)does your use case tolerate data being out of date for up to 60 seconds?

Rick•11/8/23, 5:14 PM

Can you help me understand this please? So if a user creates a new session, then the session goes into the closest kv. All subsequent requests by this user should be hitting this same kv, and as such the 60 sec latency shouldn't matter. Or am I misunderstanding this and there is no gaurantee that the user request will hit that very kv or not

kian•11/8/23, 5:15 PM

The value for a given key in a given colo (a user may hit different colos in the same point of presence, i.e London) will be cached, meaning users may see stale (or no data) despite the data being written elsewhere.

Thanks, any ETA on raising the storage per-database limit? We cannot take the risk to develop on D1

Similar Threads

Similar Threads

Similar Threads