No, see the example earlier, this has 4 categories: crushing endgame fork short 000rO,3R4/8/K7/pB2b3

No, see the example earlier, this has 4 categories: crushing endgame fork short
000rO,3R4/8/K7/pB2b3/1p6/1P2k3/3p4/8 w - - 4 58,a6a5 e5c7 a5b4 c7d8,1039,80,85,61,crushing endgame fork short,https://lichess.org/tzeeBEc2#115,

Original message was deleted

Louis DeconinckOP•2/23/23, 10:45 PM

I don't mind using an alpha product, but the CSV is 550MB.

Jjohtso @louisdck you can do some quite nice stuff using a sqlite database and static ho...

Louis DeconinckOP•2/23/23, 10:46 PM

Thanks for the suggestion, but I don't really see how I use that GitHub project together with R2 (I'm quite a newbie).

LLouis Deconinck No, see the example earlier, this has 4 categories: crushing endgame fork short ...

Hello, I’m Allie!•2/23/23, 10:46 PM

You might be able to do this with a series of extra keys? Like crushing:000r0 maps to 000r0, which maps to the row?

Louis DeconinckOP•2/23/23, 10:47 PM

Feels like reinventing a relational database

So would the value each time be one ID or an array of all ID's in that category?

ZZiga Zajc So maybe you can go with PlanetScale (If I can suggest 3rd party product)

Louis DeconinckOP•2/23/23, 10:48 PM

I can look into it, but I'm quite happy with some other CF products so was looking if they had something suiting my needs.

LLouis Deconinck Feels like reinventing a relational database 😄 So would the value each time be ...

Hello, I’m Allie!•2/23/23, 10:49 PM

The Value would either be another Key, or a row of your CSV

Louis DeconinckOP•2/23/23, 10:49 PM

So this second KV would have more than 12 million rows? 3 million games with on average 4 categories.

LLouis Deconinck Thanks for the suggestion, but I don't really see how I use that GitHub project ...

johtso•2/23/23, 10:50 PM

basically, you need to write a schema with your indexes, and import your data into a sqlite database (something like this: https://gist.github.com/johtso/c4de17de513c36f91c948498c8fc7908)

Original message was deleted

Louis DeconinckOP•2/23/23, 10:50 PM

Where can I find more about D1's restrictions?

johtso•2/23/23, 10:51 PM

@louisdck and then you specify the location and shape of the database in

createDbWorker

createDbWorker

and then you can query it using sql https://github.com/johtso/infinite-novelty-xstate/blob/master/src/machines/db.ts

LLouis Deconinck So this second KV would have more than 12 million rows? 3 million games with on ...

Hello, I’m Allie!•2/23/23, 10:52 PM

Yes, but note that other than on reads/writes, KV bills on Stored Data, not rows

Jjohtso @louisdck and then you specify the location and shape of the database in `create...

Hello, I’m Allie!•2/23/23, 10:52 PM

Does it work if you have to read in the data in chunks?

johtso•2/23/23, 10:54 PM

@HardAtWork how do you mean?

johtso•2/23/23, 10:54 PM

what do you mean by read in the data?

Jjohtso @HardAtWork how do you mean?

Hello, I’m Allie!•2/23/23, 10:55 PM

The CSV file is 550 MB, which is more than a Worker can store in memory, so you would have to read it in chunks

johtso•2/23/23, 10:56 PM

you would never read the whole thing, it's stored in R2 and you read the rows you want to show to the user, and the queries can come straight from the frontend

Jjohtso you would never read the whole thing, it's stored in R2 and you read the rows yo...

Hello, I’m Allie!•2/23/23, 10:57 PM

So it uses range requests then?

johtso•2/23/23, 10:57 PM

yep

johtso•2/23/23, 10:58 PM

and if the database file is bigger than the 512mb cache limit, or you want to trade off higher possibility of cache eviction for quicker "cold" requests, you can break the database into chunks using

create_db.sh

create_db.sh

ZZiga Zajc I have also read that you can contact them and request higher limits.

Hello, I’m Allie!•2/23/23, 11:08 PM

For Cache?

Hello, I’m Allie!•2/23/23, 11:08 PM

That’s a paid upgrade

ZZiga Zajc My bad I though there was a text that said you can contact us to increase the da...

Hello, I’m Allie!•2/24/23, 7:48 AM

For KV?

ZZiga Zajc For D1

Hello, I’m Allie!•2/24/23, 7:50 AM

Yeah, maybe at some point, but not right now

Justin Garrison•2/24/23, 6:57 PM

Is there a wrangler command or API to list all keys in a KV?

Justin Garrison•2/24/23, 7:00 PM

glad I was generally in the right area. Now trying to find my bearer token

kian•2/24/23, 7:00 PM

There's also https://developers.cloudflare.com/workers/wrangler/commands/#list-2

kian•2/24/23, 7:00 PM

For tokens, https://developers.cloudflare.com/fundamentals/api/get-started/create-token/

kian•2/24/23, 7:01 PM

I'm actually just https://github.com/arc53/DocsGPT with a Discord interface

Justin Garrison•2/24/23, 7:01 PM

oh nice! the wrangler command works great

beelzabub•2/24/23, 7:19 PM

Hi all. I apologize for neglecting this channel for a while. We've been busy for the past little while on KV 2.0 and we're getting ready to start early beta testing if you're interested. There's some architectural changes that we're going to be describing later this year in a blog post but the high level summary of the benefits are:

Eventual consistency is a lot better bounded than it is today. We've observed worst case ~days for stale reads but now it should be closer to hours and we're working on cutting that down even further.
Read after write consistency should be nominally much closer to 1 minute or less globally even when you specify a long cacheTTL.
There may be some TTFB improvements. I'd ask you please provide feedback to this thread if you do see any changes as I'm curious what people are seeing on their end.

This is still an early implementation so there's lots of improvements still planned. My hope is that over the next few months we're able to reduce the read-after-write consistency into tens of seconds. This is also a first step in terms of substantially improving TTFB (I won't spoil the surprise just yet about what number I'm targeting).

Just to be clear. Everyone is going to get this. This is purely about having an opportunity to provide feedback early - much easier for us to investigate issues when there's only a handful of accounts on this :).

Please register interest by filling out https://forms.gle/d1yrVjxpQNFSYYYf6

Google Docs

KV 2.0 Beta Sign Up

KV 2.0 will be rolling out over the next month or so. Submit interest to get into an early beta before we turn this on globally.

DDeleted User Is the *improved TTFB* for the cached reads or cold reads or both 👀

Hello, I’m Allie!•2/24/23, 7:41 PM

I assume since it is first byte it would be cold reads?

beelzabub•2/24/23, 7:56 PM

Can we move this discussion into the sign up thread? I'll answer this there

Bbeelzabub Can we move this discussion into the sign up thread? I'll answer this there

Hello, I’m Allie!•2/24/23, 7:57 PM

Is there a thread that I missed somewhere?

HHello, I’m Allie!Is there a thread that I missed somewhere?

beelzabub•2/24/23, 7:58 PM

Uggh. Maybe I didn't do it right. Let me figure out Discord

Dyno•2/24/23, 8:20 PM

Support for assets > 25 MiB in KV

OMG YES I NEED THIS

Neat

I need you to fix something else (please specify)

Poll by beelzabub#0369

•

2/24/23, 8:20 PM

DDeleted User Is the *improved TTFB* for the cached reads or cold reads or both 👀

beelzabub•2/24/23, 8:24 PM

I replied in more detail, but TLDR it should be some amount of both. "cold" is even more complicated to discuss without describing what changes we made.

Sameer Ali•2/26/23, 9:09 PM

is there anyway to gzip/br json content before storing to kv and then decode on get?

kian•2/26/23, 9:11 PM

Workers implements https://developer.mozilla.org/en-US/docs/Web/API/CompressionStream and https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream but those are things you'd have use with your own wrapper around KV - KV itself doesn't have any (de)compression options

Unsmart•2/27/23, 7:52 PM

Writes have to go to central storage so yes thats normal

zegevlier•2/28/23, 10:28 AM

Cache hits count has reads, so I would expect you to see 20 reads.

zegevlier•2/28/23, 10:29 AM

I believe so

No, see the example earlier, this has 4 categories: crushing endgame fork short 000rO,3R4/8/K7/pB2b3

Similar Threads

Similar Threads

Similar Threads