Cloudflare Developers•3y ago

Feels like reinventing a relational database 😄 So would the value each time be one ID or an array o

Feels like reinventing a relational database

So would the value each time be one ID or an array of all ID's in that category?

ZZiga Zajc So maybe you can go with PlanetScale (If I can suggest 3rd party product)

Louis DeconinckOP•2/23/23, 10:48 PM

I can look into it, but I'm quite happy with some other CF products so was looking if they had something suiting my needs.

LLouis Deconinck Feels like reinventing a relational database 😄 So would the value each time be ...

Hello, I’m Allie!•2/23/23, 10:49 PM

The Value would either be another Key, or a row of your CSV

Louis DeconinckOP•2/23/23, 10:49 PM

So this second KV would have more than 12 million rows? 3 million games with on average 4 categories.

LLouis Deconinck Thanks for the suggestion, but I don't really see how I use that GitHub project ...

johtso•2/23/23, 10:50 PM

basically, you need to write a schema with your indexes, and import your data into a sqlite database (something like this: https://gist.github.com/johtso/c4de17de513c36f91c948498c8fc7908)

Original message was deleted

Louis DeconinckOP•2/23/23, 10:50 PM

Where can I find more about D1's restrictions?

johtso•2/23/23, 10:51 PM

@louisdck and then you specify the location and shape of the database in createDbWorkercreateDbWorker and then you can query it using sql https://github.com/johtso/infinite-novelty-xstate/blob/master/src/machines/db.ts

LLouis Deconinck So this second KV would have more than 12 million rows? 3 million games with on ...

Hello, I’m Allie!•2/23/23, 10:52 PM

Yes, but note that other than on reads/writes, KV bills on Stored Data, not rows

Jjohtso @louisdck and then you specify the location and shape of the database in `create...

Hello, I’m Allie!•2/23/23, 10:52 PM

Does it work if you have to read in the data in chunks?

johtso•2/23/23, 10:54 PM

@HardAtWork how do you mean?

johtso•2/23/23, 10:54 PM

what do you mean by read in the data?

Jjohtso @HardAtWork how do you mean?

Hello, I’m Allie!•2/23/23, 10:55 PM

The CSV file is 550 MB, which is more than a Worker can store in memory, so you would have to read it in chunks

johtso•2/23/23, 10:56 PM

you would never read the whole thing, it's stored in R2 and you read the rows you want to show to the user, and the queries can come straight from the frontend

Jjohtso you would never read the whole thing, it's stored in R2 and you read the rows yo...

Hello, I’m Allie!•2/23/23, 10:57 PM

So it uses range requests then?

johtso•2/23/23, 10:57 PM

yep

johtso•2/23/23, 10:58 PM

and if the database file is bigger than the 512mb cache limit, or you want to trade off higher possibility of cache eviction for quicker "cold" requests, you can break the database into chunks using create_db.shcreate_db.sh

ZZiga Zajc I have also read that you can contact them and request higher limits.

Hello, I’m Allie!•2/23/23, 11:08 PM

For Cache?

Hello, I’m Allie!•2/23/23, 11:08 PM

That’s a paid upgrade

ZZiga Zajc My bad I though there was a text that said you can contact us to increase the da...

Hello, I’m Allie!•2/24/23, 7:48 AM

For KV?

ZZiga Zajc For D1

Hello, I’m Allie!•2/24/23, 7:50 AM

Yeah, maybe at some point, but not right now

Justin Garrison•2/24/23, 6:57 PM

Is there a wrangler command or API to list all keys in a KV?

Justin Garrison•2/24/23, 7:00 PM

glad I was generally in the right area. Now trying to find my bearer token

kian•2/24/23, 7:00 PM

There's also https://developers.cloudflare.com/workers/wrangler/commands/#list-2

kian•2/24/23, 7:00 PM

For tokens, https://developers.cloudflare.com/fundamentals/api/get-started/create-token/

kian•2/24/23, 7:01 PM

I'm actually just https://github.com/arc53/DocsGPT with a Discord interface

Justin Garrison•2/24/23, 7:01 PM

oh nice! the wrangler command works great

beelzabub•2/24/23, 7:19 PM

Hi all. I apologize for neglecting this channel for a while. We've been busy for the past little while on KV 2.0 and we're getting ready to start early beta testing if you're interested. There's some architectural changes that we're going to be describing later this year in a blog post but the high level summary of the benefits are:

Eventual consistency is a lot better bounded than it is today. We've observed worst case ~days for stale reads but now it should be closer to hours and we're working on cutting that down even further.
Read after write consistency should be nominally much closer to 1 minute or less globally even when you specify a long cacheTTL.
There may be some TTFB improvements. I'd ask you please provide feedback to this thread if you do see any changes as I'm curious what people are seeing on their end.

This is still an early implementation so there's lots of improvements still planned. My hope is that over the next few months we're able to reduce the read-after-write consistency into tens of seconds. This is also a first step in terms of substantially improving TTFB (I won't spoil the surprise just yet about what number I'm targeting).

Just to be clear. Everyone is going to get this. This is purely about having an opportunity to provide feedback early - much easier for us to investigate issues when there's only a handful of accounts on this :).

Please register interest by filling out https://forms.gle/d1yrVjxpQNFSYYYf6

Google Docs

KV 2.0 Beta Sign Up

KV 2.0 will be rolling out over the next month or so. Submit interest to get into an early beta before we turn this on globally.

DDeleted User Is the *improved TTFB* for the cached reads or cold reads or both 👀

Hello, I’m Allie!•2/24/23, 7:41 PM

I assume since it is first byte it would be cold reads?

beelzabub•2/24/23, 7:56 PM

Can we move this discussion into the sign up thread? I'll answer this there

Bbeelzabub Can we move this discussion into the sign up thread? I'll answer this there

Hello, I’m Allie!•2/24/23, 7:57 PM

Is there a thread that I missed somewhere?

HHello, I’m Allie!Is there a thread that I missed somewhere?

beelzabub•2/24/23, 7:58 PM

Uggh. Maybe I didn't do it right. Let me figure out Discord

Dyno•2/24/23, 8:20 PM

Support for assets > 25 MiB in KV

OMG YES I NEED THIS

Neat

I need you to fix something else (please specify)

Poll by beelzabub#0369

•

2/24/23, 8:20 PM

DDeleted User Is the *improved TTFB* for the cached reads or cold reads or both 👀

beelzabub•2/24/23, 8:24 PM

I replied in more detail, but TLDR it should be some amount of both. "cold" is even more complicated to discuss without describing what changes we made.

Sameer Ali•2/26/23, 9:09 PM

is there anyway to gzip/br json content before storing to kv and then decode on get?

kian•2/26/23, 9:11 PM

Workers implements https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream and https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream but those are things you'd have use with your own wrapper around KV - KV itself doesn't have any (de)compression options

Unsmart•2/27/23, 7:52 PM

Writes have to go to central storage so yes thats normal

zegevlier•2/28/23, 10:28 AM

Cache hits count has reads, so I would expect you to see 20 reads.

zegevlier•2/28/23, 10:29 AM

I believe so

Jadugar_Jaggu•3/3/23, 11:41 AM

I request to Make a Realtime Key-Value storage model, like Firebase real-time database.

JJadugar_Jaggu I request to Make a Realtime Key-Value storage model, like Firebase real-time da...

beelzabub•3/3/23, 6:14 PM

For KV, requests for the same key are satisfied within milliseconds and it doesn't matter where in the world you access this from. My understanding is that real-time databases have access time measured in hundreds of milliseconds and requests from other parts of the world are slower. Real time databases are great tech, but it's not comparable to the kind of storage KV offers (totally different pricing, totally different usage patterns optimized for, totally different invariants in terms of API).

If you can use KV, I think you'll see more value out of KV (cheaper & faster). It won't be suitable for every problem so YMMV. R2 is strongly consistent like firebase and will be cheaper than firebase, but it won't offer the database-like features if you need them (e.g. transactions).

It can sometimes be more helpful to describe your use-case @Jadugar_Jaggu rather than asking for a specific product. If you can help me understand the problem you're using a realtime database to solve for, it's possible there are other ways to accomplish that. If you're looking for a drop in replacement for Firebase, I don't think CF has anything like that at this time to talk about.

Bbeelzabub For KV, requests for the same key are satisfied within milliseconds and it doesn...

Jadugar_Jaggu•3/4/23, 7:30 AM

I am using KV on my service which has a total of 15 million traffic per month, It is working well, and Seriously KV helps me to achieve a 60% Profit margin from my business by lowering the various servers and management costs.

I am a KV user for 9 months approx

Now I was thinking of making a realtime application like you know about Trading and Fantasy Sports apps in which a large number of users perform action simultaneously.

Jadugar_Jaggu•3/4/23, 7:31 AM

Thats Why I just requested if KV can be helpful in realtime database model it can save lots of expenses

Jadugar_Jaggu•3/4/23, 7:37 AM

And Seriously I don't want to know how Cloudflare manages to lower the costs while delivering good performance, I just enjoy the profits

Feels like reinventing a relational database 😄 So would the value each time be one ID or an array o

Similar Threads