U- Batch Write API (In workers as it exists on CF API)
U- Streams/Worker triggers
Whenever a value is created/updated/deleted run a worker script
I- prewarm - i.e. I'm going to use these keys on the next request or shortly, please cache them in this colo (but don't do a get because of RAM limits)
D- larger metadata size limit (eg. 8-16KB)
D- list with values (maybe limit of 1MB data combined)
U- Another region (Asia)
[deno has this for example, faster reads for eventual consistency around the world]
PHello, Is there a way to delete all of the keys in a KV namespace at once? Similar to FLUSHDB in redis?
LI'm trying to reason about the performance of list() operations using the prefix option to make decisions for how to model a many to many relationship in KV. I could implement it several ways and performance test, but I'm trying to avoid that premature optimization work. A Person can be part of several Orgs and each Org can have many Persons (upwards of 10,000). Assume two Persons with ids “p1" and "p2" and two Orgs with ids “o1" and "o2". Also, assume, eventual consistency is fine for my use case.
Option 1 just keys:
"person/p1/org/o1"
“person/p1/org/o2”
"person/p2/org/o1"
“org/o1/person/p1”
“org/o1/person/p2”
“org/o2/person/p1”
Then, when I wanted to find all of Orgs that go with p1, I could use prefix “person/p1”. That would get me back two rows. I could parse those keys and extract the ids for those two orgs. Similarly, if I wanted to find all of the Persons that go with Org o2, I could use prefix “org/o2”, parse that to find the one person that goes with it. The operations for adding or removing a relationship only ever effects two keys so those changes will be cheap but what about the list() operations when there are 10,000 Persons in one Org? If the keys are stored in an index that is kept in memory, I’m guessing that will also be fine.
Option 2 big and ever growing values:
I would have just one key/value pair for each Person and one for each Org. The value would contain a serialized array of the values of the other side of the relationship. Query would be fast, but changes would require reading, deserializing, filtering or adding-to, re-serializing, and finally saving a potentially large array.
Thoughts?
Option 1 just keys:
"person/p1/org/o1"
“person/p1/org/o2”
"person/p2/org/o1"
“org/o1/person/p1”
“org/o1/person/p2”
“org/o2/person/p1”
Then, when I wanted to find all of Orgs that go with p1, I could use prefix “person/p1”. That would get me back two rows. I could parse those keys and extract the ids for those two orgs. Similarly, if I wanted to find all of the Persons that go with Org o2, I could use prefix “org/o2”, parse that to find the one person that goes with it. The operations for adding or removing a relationship only ever effects two keys so those changes will be cheap but what about the list() operations when there are 10,000 Persons in one Org? If the keys are stored in an index that is kept in memory, I’m guessing that will also be fine.
Option 2 big and ever growing values:
I would have just one key/value pair for each Person and one for each Org. The value would contain a serialized array of the values of the other side of the relationship. Query would be fast, but changes would require reading, deserializing, filtering or adding-to, re-serializing, and finally saving a potentially large array.
Thoughts?
BThat’s a fantastic question. I’ll note that option 2 also has a challenge in that you’ll need to serialize updates in some way and even then you risk corrupting your store if you don’t have a more sequential record (we don’t offer conditional writes currently). I think the main question is what are the most common operations you’ll want to do with these? I don’t know your problem domain but listing 10k items in a foreground operation feels like a misdesign and the cost in a background operation won’t be that much (reminder that you can only list 1k at a time)
MCan KV do conditional writes and/or transactions?
HIt cannot
JAPI and/or paginated Worker is really your only way today. You'll just have to slowly iterate through keys to avoid hitting any limits and such
MI added opentelemetry tracing and can see that when I do lots of parallel reads/writes with KV (25+), I'm hitting the 6 concurrent connections limit. Something I was hoping would take ~250ms is taking about 1.2s+. I wonder if perhaps I am using the wrong solution? My origin is returning multiple entities that I want to cache in KV separately
PHello! I'm curious if there is a way to delete all of the keys in a KV namespace at once? Similar to FLUSHDB in redis? I know there is a batch process where you select multiple keys at once an delete them but I was wondering if there was a way to do this in one command without recreating the namespace.
Thanks!
Thanks!
UOne more feature I forgot about that would be nice:
- Reverse listing items
Will make it so I can sort date based ids how I want them
HKV doesn’t currently support that
Poh I see, so the only current option is to pull all of the keys and delete them?
Mwhen running workers-rs with wrangler on local mode, my code attempts to store a value on the key value database, (no errors), but when i check the dashboard on cloudflare, there doesn't seem to actually be anything stored
Manyone know why?
CLocal mode doesn’t effect any remote resources
Mah i see
Mso if i use on my worker with local mode enabled and attempt to add something, it'll just be stored somewhere locally on my system?
COnly if you have the persist flag (wrangler 2). Otherwise it is stored in memory and is gone once dev stops
Malright, thank you
BDoes KV support secondary indexes?
INo.
IYou can list keys by prefix but that's it.
BThanks!
BSo to "resort" something I'd have to delete and readd it with a new key right? @Isaac McFadyen | YYZ01
IYeah pretty much.
BCoolio
IKeep in mind that because KV is eventually consistent if you read a value then change the value with the same key it might take up to 60s to see the new value in other locations
IIf you change the key, as long as you haven't queried the key before, it should be instant.
IIf you've queried the key before, even if it didn't exist at the time, it may take up to 60s to see the new value
BHuh... Nah, this key will have been queried before 
IIn which case, yeah, 60s (or more in very rare cases) to see in other locations.
BIt'll be queried and then immediately updated again
BFair enough, thanks!
B@Isaac McFadyen | YYZ01 do you know if there's a way to count keys? Something like the
list()I don't believe so, no, aside from just manually calling
listI'll have to do that then. I know that there's eventual consistency involved, but I'll need a count even if it's a rough count
Ilist