You could do background revalidation though, if that fits better?

HHard@Work You could do background revalidation though, if that fits better?

Stoke•7/26/24, 11:42 AM

Thanks for the clarification! What exactly do you mean with doing background revalidation? I was under the assumption that KV is doing that already? From https://developers.cloudflare.com/kv/reference/how-kv-works/: "reads are periodically revalidated in the background"

SStoke Thanks for the clarification! What exactly do you mean with doing background rev...

Hard@WorkOP•7/26/24, 11:43 AM

Uh, that part may require an update? There was a thing called KV v2, which was supposed to do background revalidation, but iirc that was rolled back, and it doesn't happen anymore

Stoke•7/26/24, 11:43 AM

I read that plus the note about the "60s for consistency" as "background revalidation can take up to 60s"

Hard@WorkOP•7/26/24, 11:43 AM

Though let me check

SStoke I read that plus the note about the "60s for consistency" as "background revalid...

thomasgauvin•7/26/24, 6:55 PM

KV does indeed perform revalidation at some point before the cacheTtl is elapsed. We'll continue keeping the docs up to date, but as of now, they are accurate.

SStoke Hi friends, a question re visibility of writes. The docs are not fully clear to ...

thomasgauvin•7/26/24, 7:15 PM

When you write to KV, the changes are immediately made to KV's origin stores. They are also immediately visible in the region where the KV write came through, regardless of cacheTtl. However, we recommend you do not rely on this behavior because in rare instances your request to your Worker may get routed to another location which will not have this change immediately visible. This is why we recommend using Durable Objects for atomic operations.

In other regions, it is possible that a stale value has been read & cached in a location before the write has been made. In this case, the change will only be visible in that location when the cache in that region elapses and is revalidated.

Marcus•7/27/24, 6:19 PM

Is it (still?) true that there's no way to move a KV namespace to another Cloudflare account besides writing a script to bulk-download every key/value pair from one and bulk-upload to another?

Marcus•7/27/24, 7:03 PM

(and that would consume 1 write per key, so on the free tier this would have to be spread over ~keys/1000keys/1000 days?)

Marcus•7/27/24, 7:26 PM

I see a bulk write endpoint, but it only takes JSON and our values are binary so that doesn't help

Walshy•7/27/24, 8:18 PM

you can provide base64

Walshy•7/27/24, 8:19 PM

https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-multiple-key-value-pairs

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Walshy•7/27/24, 8:19 PM

base64
boolean
Whether or not the server should base64 decode the value before storing it. Useful for writing values that wouldn't otherwise be valid JSON strings, such as images.

Marcus•7/27/24, 9:07 PM

ooh, interesting. I guess I could download one-by-one, convert to base64 locally, bulk upload. do you know if that endpoint counts as a single write re: the daily limits?

Walshy•7/27/24, 9:11 PM

https://developers.cloudflare.com/kv/platform/pricing/#frequently-asked-questions

Each key-value pair in the (bulk write) PUT request is counted as a single write, identical to how each call to PUT in the Workers API counts as a write. Writing 5,000 keys via the REST API incurs the same write costs as making 5,000 PUT calls in a Worker.

John Spurlock•7/28/24, 10:35 PM

nice! new KV storage and operations datasets available in the cf graphql api: kvOperationsAdaptiveGroupskvOperationsAdaptiveGroups and kvStorageAdaptiveGroupskvStorageAdaptiveGroups

was always a bit strange having to go the rest api for those

MMarcus ooh, interesting. I guess I could download one-by-one, convert to base64 locally...

Marcus•7/28/24, 11:05 PM

resulting (somewhat hacky) script, in case someone else needs to do this: https://github.com/marcustyphoon/cloudflare-kv-migrate

Tthomasgauvin When you write to KV, the changes are immediately made to KV's origin stores. Th...

Stoke•7/29/24, 8:46 AM

I see, thanks for the info. Are durable objects really a viable replacement for KV? To be a bit more concrete on the use case; let's imagine a link shortener service with 100k links (per month) and 20M clicks per month. My real/actual use case is close enough to this example scenario. I would store link id -> link URL as key -> value. The load is very read heavy with spikes of over 200 rps, but there are only very few updates to existing keys, let's say 1 per hour. I would traditionally use Redis (or similar) for such a use case, that's why I'm looking at KV. Using durable objects, this would be a single object for the entire map of links? Or one object per link (i.e. 100k objects per month)? Since duration is billed, would the latter mean that 100k objects per month are "active" indefinitely?

SStoke I see, thanks for the info. Are durable objects really a viable replacement for ...

Hard@WorkOP•7/29/24, 8:50 AM

No, they are only billable when processing requests, even if they are still "active"

Stoke•7/29/24, 9:35 AM

Ok thanks. Every request activates for 30s, if I understand correctly. Makes sense. Then after reading through the docs I can see that the recommendationis to not use single global objects, so I think "object per link entity" would be the way to go here. I still struggle to wrap my head around why/how Durable Objects are the best fit for this use case. They're certainly cool and powerful, but I don't really need all the extra guarantees. All I'd really need is a bit of a narrower windows for eventual consistency in KV. I think not having a TTL and the first lookup being "a bit slow" are both real downsides for me, albeit both would be acceptable. I'm mainly worried about the complexity here... KV is very simple to grasp, whereas DO is quite complex conceptually, even when misused as a key-value store

Zack•7/29/24, 11:22 AM

Hey, I am seeing some pretty horrendous latencies using workers with KV, requests to get a 3.6MB value are taking up to 10 seconds at times, this is from a london IP and the KV was created by me from an IP in the UK too. Not sure why sometimes the latencies are so bad, I was using this to speed up a bunch of queries that we would usually make to R2 as range requests, but it appears sometimes KV is substantially slower? Any ideas what could be going on here?

Zack•7/29/24, 11:24 AM

These requests are hitting the LHR colo, latencies for R2 also seem to be fluctuating pretty badly for these same requests

Zack•7/29/24, 11:26 AM

Maybe it's just an issue ongoing at that particular colo? Requests from other parts of the world such as Toronto, Berlin, Kyiv are all responding fast

thomasgauvin•7/29/24, 9:42 PM

Thanks for the details @Zack that you shared in DMs, we'll keep our eyes open for LHR (UK region) latencies if it returns and investigate in our past logs.

Jay•7/30/24, 1:14 AM

I'm seeing some strange behavior around my KV right now, where for some reason deleted data is persisting.
I've attached two screenshots of what's going on (empty KV but log stating there is data there):
This is the snippet:

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

Jay•7/30/24, 1:15 AM

Is this a bug? env.<namespace>.list()env.<namespace>.list() correctly shows nothing, but for this key it has persisted across multiple reloads, not to mention operations that used to work now just silently failing (putput, for one that I noticed)

Jay•7/30/24, 1:25 AM

I don't know if its some sort of race condition considering I'm doing multiple KV operations across different namespaces. Is there a "safe" way to put/delete data that I'm missing?

Chaika•7/30/24, 1:25 AM

KV's eventually consistent, both the dash and list go back to one of the central stores irrc. Are you overriding cache ttl or just using the default?

Jay•7/30/24, 1:25 AM

I don't have any cache set, so probably default

Jay•7/30/24, 1:26 AM

Have never seen a delay in consistency this long though

Chaika•7/30/24, 1:26 AM

default is just 60s

Jay•7/30/24, 1:26 AM

This has gone over 10 minutes

Jay•7/30/24, 1:26 AM

still happening with this key

Jay•7/30/24, 1:27 AM

same log new timestamp to show ^

Jay•7/30/24, 1:30 AM

The value isn't overly large or anything, its just a stringified object about as long as this message

Chaika•7/30/24, 1:31 AM

yea, weird. I don't see any issues, just updated and deleted a key to test. You said PUTs were failing as well, to the same key?

Jay•7/30/24, 1:33 AM

Yeah so I had a line that looked like this to test (and I'm aware I believe just putting overwrites but I was testing something iirc):

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

which:
- showed the key as deleted in the CF dashboard
- did not run the putput according to the dashboard
- key is still retrievable

Jay•7/30/24, 1:34 AM

Let me check if the put went through just programmatically

Jay•7/30/24, 1:35 AM

Ok yeah the put went through programmatically its just not updating on the CF dash

Jay•7/30/24, 1:37 AM

agh so it seems like things are fine internally the dashboard is just never to be trusted

Anthony•8/2/24, 9:43 AM

How do you modify every KV entry without being ratelimited?

"Error: Too many API requests by single worker invocation."

Ben•8/2/24, 5:21 PM

Ok so I have a question regarding the KV list and specifically the cursor.

I want to list all values of a workers KV store. I am using a cursor to "go through" the entire KV store. What happens if a new entry is created with a brand new key? Would I need to list through the KV store again or is it guaranteed to exist at the end of my pagination? Thanks!

jason•8/2/24, 7:22 PM

I'm using Cloudflare Pages with KV.

My goal is to set up KV for different environments, to have:
1. one KV id for production, and
2. another KV id for all other environments (preview URLs, local dev with miniflare maybe, etc)

The docs show two approaches:

v1

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

OR

v2 (using environments)

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

- which is preferred and why?
- for the first version, how does Cloudflare differentiate when to use the KV represented by

id

id

preview_id

preview_id

kian•8/2/24, 7:23 PM

preview_id

preview_id

is related to wrangler devwrangler dev, not environments

jason•8/2/24, 7:24 PM

say more please

Abakir•8/7/24, 5:37 PM

what is the purpose of metadata for key value pairs cloudflare? Why would you use that verses just using a value?

AAbakir what is the purpose of metadata for key value pairs cloudflare? Why would you us...

Isaac McFadyen•8/7/24, 5:38 PM

Metadata can be retrieved using a

list

list

call, values cannot.

Isaac McFadyen•8/7/24, 5:39 PM

So if you have a couple thousand keys, and store important data in the metadata, you can get all that data with one call versus iterating through and using 1000+ calls (and being billed for 1000+ calls)

IIsaac McFadyen So if you have a couple thousand keys, and store important data in the metadata,...

Abakir•8/7/24, 5:40 PM

This makes sense thank you

jacob•8/7/24, 9:55 PM

Anybody who's managed to use KV with container based DI like tsyringe? I find that the kv namespace methods break once put through tsyringe. Specifically in classes using it as a dependency. I get super weird behaviour where the methods don't throw any sort of error but seemingly just stop any execution of code coming after it.

jacob•8/7/24, 10:27 PM

It actually appears to be unrelated to the container. I'm seeing the same behaviour simply passing it down a constructor, the kv namespace methods become unusable.

You could do background revalidation though, if that fits better?

Similar Threads

Similar Threads

Similar Threads