@louisdck you can do some quite nice stuff using a sqlite database and static hosting https://github
@louisdck you can do some quite nice stuff using a sqlite database and static hosting https://github.com/phiresky/sql.js-httpvfs
JI'm currently hosting 4mil-ish rows with 3 or 4 indexes on R2, and the whole thing fits in cloudflare's cache
Jdefinitely easy to get uploaded as it's just a single file
LMostly ID, but as mentioned it would be nice only to return records within a specific category.
LOk, what else would you recommend?
Jsqlite is perfect for large read-only datasets
HDoes each record only have one category?
JI feel like 3mil rows containing chess puzzles is probably going to be more than a GB
Jit's like a game of guess the number of sweets in the jar 
LNo, see the example earlier, this has 4 categories: crushing endgame fork short
000rO,3R4/8/K7/pB2b3/1p6/1P2k3/3p4/8 w - - 4 58,a6a5 e5c7 a5b4 c7d8,1039,80,85,61,crushing endgame fork short,https://lichess.org/tzeeBEc2#115,
000rO,3R4/8/K7/pB2b3/1p6/1P2k3/3p4/8 w - - 4 58,a6a5 e5c7 a5b4 c7d8,1039,80,85,61,crushing endgame fork short,https://lichess.org/tzeeBEc2#115,
LI don't mind using an alpha product, but the CSV is 550MB.
LThanks for the suggestion, but I don't really see how I use that GitHub project together with R2 (I'm quite a newbie).
HYou might be able to do this with a series of extra keys? Like crushing:000r0 maps to 000r0, which maps to the row?
LFeels like reinventing a relational database 
So would the value each time be one ID or an array of all ID's in that category?
So would the value each time be one ID or an array of all ID's in that category?
LI can look into it, but I'm quite happy with some other CF products so was looking if they had something suiting my needs.
HThe Value would either be another Key, or a row of your CSV
LSo this second KV would have more than 12 million rows? 3 million games with on average 4 categories.
Jbasically, you need to write a schema with your indexes, and import your data into a sqlite database (something like this: https://gist.github.com/johtso/c4de17de513c36f91c948498c8fc7908)
LWhere can I find more about D1's restrictions?
J@louisdck and then you specify the location and shape of the database in
createDbWorker and then you can query it using sql https://github.com/johtso/infinite-novelty-xstate/blob/master/src/machines/db.tsHYes, but note that other than on reads/writes, KV bills on Stored Data, not rows
HDoes it work if you have to read in the data in chunks?
J@HardAtWork how do you mean?
Jwhat do you mean by read in the data?
HThe CSV file is 550 MB, which is more than a Worker can store in memory, so you would have to read it in chunks
Jyou would never read the whole thing, it's stored in R2 and you read the rows you want to show to the user, and the queries can come straight from the frontend
HSo it uses range requests then?
Jyep
Jand if the database file is bigger than the 512mb cache limit, or you want to trade off higher possibility of cache eviction for quicker "cold" requests, you can break the database into chunks using
create_db.shHFor Cache?
HThat’s a paid upgrade
HYeah, maybe at some point, but not right now
JIs there a wrangler command or API to list all keys in a KV?
Jglad I was generally in the right area. Now trying to find my bearer token
KKKJoh nice! the wrangler command works great
BHi all. I apologize for neglecting this channel for a while. We've been busy for the past little while on KV 2.0 and we're getting ready to start early beta testing if you're interested. There's some architectural changes that we're going to be describing later this year in a blog post but the high level summary of the benefits are:
Just to be clear. Everyone is going to get this. This is purely about having an opportunity to provide feedback early - much easier for us to investigate issues when there's only a handful of accounts on this :).
Please register interest by filling out https://forms.gle/d1yrVjxpQNFSYYYf6
- Eventual consistency is a lot better bounded than it is today. We've observed worst case ~days for stale reads but now it should be closer to hours and we're working on cutting that down even further.
- Read after write consistency should be nominally much closer to 1 minute or less globally even when you specify a long cacheTTL.
- There may be some TTFB improvements. I'd ask you please provide feedback to this thread if you do see any changes as I'm curious what people are seeing on their end.
Just to be clear. Everyone is going to get this. This is purely about having an opportunity to provide feedback early - much easier for us to investigate issues when there's only a handful of accounts on this :).
Please register interest by filling out https://forms.gle/d1yrVjxpQNFSYYYf6
Google Docs
KV 2.0 will be rolling out over the next month or so. Submit interest to get into an early beta before we turn this on globally.
HI assume since it is first byte it would be cold reads?
BCan we move this discussion into the sign up thread? I'll answer this there
HIs there a thread that I missed somewhere?
BUggh. Maybe I didn't do it right. Let me figure out Discord
DSupport for assets > 25 MiB in KV
OMG YES I NEED THIS
Neat
I need you to fix something else (please specify)Poll by beelzabub#0369
•2/24/23, 8:20 PM
