Is it recommended to re-use the KV storage in multiple workers? Or should be create multiple KV ? F

Is it recommended to re-use the KV storage in multiple workers? Or should be create multiple KV ?

For example I have one KV storage :

KVCACHE

KVCACHE

KVCACHE

KVCACHE and re-using in 4-5 workers to cache the basic info using different keys. Is it ok?

or any performance improvement is expected if I create multiple cache storage for each worker as they are running on different region using smart placement binding with D1 APAC, EUROPE etc.

VVikash Is it recommended to re-use the KV storage in multiple workers? Or should be cre...

Hello, I’m Allie!•7/15/24, 7:18 AM

All KV instances share the same regions, so making a new namespace is mostly just a visual representation

HHello, I’m Allie!All KV instances share the same regions, so making a new namespace is mostly jus...

VikashOP•7/15/24, 7:27 AM

So, re-using is good.

VVikash So, re-using is good.

Hello, I’m Allie!•7/15/24, 7:37 AM

Yeah, it’s fine

Coffee11•7/16/24, 6:47 AM

▲ [WARNING] Processing wrangler.toml configuration:

- Unexpected fields found in browser field: "kv_namespaces"

kv_namespaces = [
{ binding = "CRAWLER_SCREENSHOTS_KV", id = "test" },
{ binding = "CRAWLER_LINKS_KV", id = "test" }
]

what is the correct way?

akazwz•7/16/24, 8:53 AM

@Coffee11 [[kv_namespaces]]
binding = "CRAWLER_SCREENSHOTS_KV"
id = "test"

[[kv_namespaces]]
binding = "CRAWLER_LINKS_KV"
id = "test"

Disciple of Caoximeihao•7/17/24, 5:03 AM

help

Disciple of Caoximeihao•7/17/24, 7:32 AM

how to used kv in production

Disciple of Caoximeihao•7/17/24, 7:33 AM

in localhost

DDisciple of Caoximeihao how to used kv in production

Hello, I’m Allie!•7/17/24, 7:36 AM

Use —remote—remote

DDisciple of Caoximeihao help

akazwz•7/17/24, 7:56 AM

add preview_id will work. [[kv_namespaces]]
binding = "KV"
id = "xxxx"
preview_id = "xxxx"

Julian•7/18/24, 10:14 PM

Hello. Where or how can I generate the KV API Key and Authorization Token? I will use the API so yeah I need those things.

Julian•7/18/24, 10:14 PM

Using CF Dash I can only generate the API Key, not the Auth token

JJulian Hello. Where or how can I generate the KV API Key and Authorization Token? I wil...

Cyb3r-Jak3•7/18/24, 10:18 PM

There are two ways to use the API, either your email and global token or an API token (recommended).

Julian•7/18/24, 10:25 PM

So the API Token would be generated by going to the profile -> API Keys -> KV read and KV edit, right?

JJulian So the API Token would be generated by going to the profile -> API Keys -> KV re...

Cyb3r-Jak3•7/18/24, 10:41 PM

Correct

anhhoang•7/24/24, 8:03 AM

Hello, When I query the value of a key in Kv, the result returned is null even though the value is not null. I have tested the REST API, and when running it locally, the results are correct. However, the issue occurs only when the application is deployed and the API is called on the worker. Please help me !

Aanhhoang Hello, When I query the value of a key in Kv, the result returned is null even t...

Hard@Work•7/24/24, 8:05 AM

The KV API may have cached a null response. Try to give it a minute or two, and see if the issue persists

anhhoang•7/24/24, 8:09 AM

i have data in Kv before call api and it not works

anhhoang•7/24/24, 8:10 AM

I have been encountering this issue with this key for the past 5 hours.

anhhoang•7/24/24, 8:12 AM

It works correctly when I remote from local. The issue only occurs when deployed.

Aanhhoang It works correctly when I remote from local. The issue only occurs when deployed...

thomasgauvin•7/25/24, 3:11 PM

what is your configuration in wrangler.toml? you mention using

wrangler dev --remote

wrangler dev --remote

. That will be using the preview id of your kv_namespace configuration. If you want your deployed worker to use the same KV store, you would need to have the same namespace id for both id and preview_id. Is that the case?

ex wrangler.toml

[[kv_namespaces]]
binding = "assets"
id = "<some-id>"
preview_id = "<some-id-2>"

Stoke•7/26/24, 7:40 AM

Hi friends, a question re visibility of writes. The docs are not fully clear to me in this regard. When I make an update to a value in KV, does the visibility of the change depend on the cacheTtlcacheTtl value? According to the docs, changes are immediately visible in the same location and take up to 60s in other locations. Is that also true for already cached values with high cacheTtlcacheTtl ? I would like to use 24h24h, but the (rare) changes need to be visible faster than that. A few minutes is acceptable.
I did some rough testing and it seems like cacheTtlcacheTtl does not matter, but I just want to make sure. Thank you!

Yu•7/26/24, 10:39 AM

Hi team, regarding write visibility, I would like to know approximately how long it takes for global data synchronization. Right now, after writing to KV and then reading, I basically have to wait for an additional 20 seconds to obtain the data.

YYu Hi team, regarding write visibility, I would like to know approximately how long...

Hard@Work•7/26/24, 10:41 AM

Around 60 seconds by default, but if you set a cacheTtlcacheTtl, then whatever value you set it to(minimum 60 seconds)

SStoke Hi friends, a question re visibility of writes. The docs are not fully clear to ...

Hard@Work•7/26/24, 10:42 AM

For global updates, there isn't really a good way to cache-bust them. Assuming that you read with the same cacheTtlcacheTtl value every time, it would show stale values for up to 24 hours

Hard@Work•7/26/24, 10:42 AM

You could do background revalidation though, if that fits better?

HHard@Work You could do background revalidation though, if that fits better?

Stoke•7/26/24, 11:42 AM

Thanks for the clarification! What exactly do you mean with doing background revalidation? I was under the assumption that KV is doing that already? From https://developers.cloudflare.com/kv/reference/how-kv-works/: "reads are periodically revalidated in the background"

SStoke Thanks for the clarification! What exactly do you mean with doing background rev...

Hard@Work•7/26/24, 11:43 AM

Uh, that part may require an update? There was a thing called KV v2, which was supposed to do background revalidation, but iirc that was rolled back, and it doesn't happen anymore

Stoke•7/26/24, 11:43 AM

I read that plus the note about the "60s for consistency" as "background revalidation can take up to 60s"

Hard@Work•7/26/24, 11:43 AM

Though let me check

SStoke I read that plus the note about the "60s for consistency" as "background revalid...

thomasgauvin•7/26/24, 6:55 PM

KV does indeed perform revalidation at some point before the cacheTtl is elapsed. We'll continue keeping the docs up to date, but as of now, they are accurate.

SStoke Hi friends, a question re visibility of writes. The docs are not fully clear to ...

thomasgauvin•7/26/24, 7:15 PM

When you write to KV, the changes are immediately made to KV's origin stores. They are also immediately visible in the region where the KV write came through, regardless of cacheTtl. However, we recommend you do not rely on this behavior because in rare instances your request to your Worker may get routed to another location which will not have this change immediately visible. This is why we recommend using Durable Objects for atomic operations.

In other regions, it is possible that a stale value has been read & cached in a location before the write has been made. In this case, the change will only be visible in that location when the cache in that region elapses and is revalidated.

Marcus•7/27/24, 6:19 PM

Is it (still?) true that there's no way to move a KV namespace to another Cloudflare account besides writing a script to bulk-download every key/value pair from one and bulk-upload to another?

Marcus•7/27/24, 7:03 PM

(and that would consume 1 write per key, so on the free tier this would have to be spread over ~keys/1000keys/1000 days?)

Marcus•7/27/24, 7:26 PM

I see a bulk write endpoint, but it only takes JSON and our values are binary so that doesn't help

Walshy•7/27/24, 8:18 PM

you can provide base64

Walshy•7/27/24, 8:19 PM

https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-multiple-key-value-pairs

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Walshy•7/27/24, 8:19 PM

base64
boolean
Whether or not the server should base64 decode the value before storing it. Useful for writing values that wouldn't otherwise be valid JSON strings, such as images.

Marcus•7/27/24, 9:07 PM

ooh, interesting. I guess I could download one-by-one, convert to base64 locally, bulk upload. do you know if that endpoint counts as a single write re: the daily limits?

Walshy•7/27/24, 9:11 PM

https://developers.cloudflare.com/kv/platform/pricing/#frequently-asked-questions

Each key-value pair in the (bulk write) PUT request is counted as a single write, identical to how each call to PUT in the Workers API counts as a write. Writing 5,000 keys via the REST API incurs the same write costs as making 5,000 PUT calls in a Worker.

John Spurlock•7/28/24, 10:35 PM

nice! new KV storage and operations datasets available in the cf graphql api: kvOperationsAdaptiveGroupskvOperationsAdaptiveGroups and kvStorageAdaptiveGroupskvStorageAdaptiveGroups

was always a bit strange having to go the rest api for those

MMarcus ooh, interesting. I guess I could download one-by-one, convert to base64 locally...

Marcus•7/28/24, 11:05 PM

resulting (somewhat hacky) script, in case someone else needs to do this: https://github.com/marcustyphoon/cloudflare-kv-migrate

Tthomasgauvin When you write to KV, the changes are immediately made to KV's origin stores. Th...

Stoke•7/29/24, 8:46 AM

I see, thanks for the info. Are durable objects really a viable replacement for KV? To be a bit more concrete on the use case; let's imagine a link shortener service with 100k links (per month) and 20M clicks per month. My real/actual use case is close enough to this example scenario. I would store link id -> link URL as key -> value. The load is very read heavy with spikes of over 200 rps, but there are only very few updates to existing keys, let's say 1 per hour. I would traditionally use Redis (or similar) for such a use case, that's why I'm looking at KV. Using durable objects, this would be a single object for the entire map of links? Or one object per link (i.e. 100k objects per month)? Since duration is billed, would the latter mean that 100k objects per month are "active" indefinitely?

SStoke I see, thanks for the info. Are durable objects really a viable replacement for ...

Hard@Work•7/29/24, 8:50 AM

No, they are only billable when processing requests, even if they are still "active"

Stoke•7/29/24, 9:35 AM

Ok thanks. Every request activates for 30s, if I understand correctly. Makes sense. Then after reading through the docs I can see that the recommendationis to not use single global objects, so I think "object per link entity" would be the way to go here. I still struggle to wrap my head around why/how Durable Objects are the best fit for this use case. They're certainly cool and powerful, but I don't really need all the extra guarantees. All I'd really need is a bit of a narrower windows for eventual consistency in KV. I think not having a TTL and the first lookup being "a bit slow" are both real downsides for me, albeit both would be acceptable. I'm mainly worried about the complexity here... KV is very simple to grasp, whereas DO is quite complex conceptually, even when misused as a key-value store

Zack•7/29/24, 11:22 AM

Hey, I am seeing some pretty horrendous latencies using workers with KV, requests to get a 3.6MB value are taking up to 10 seconds at times, this is from a london IP and the KV was created by me from an IP in the UK too. Not sure why sometimes the latencies are so bad, I was using this to speed up a bunch of queries that we would usually make to R2 as range requests, but it appears sometimes KV is substantially slower? Any ideas what could be going on here?

Zack•7/29/24, 11:24 AM

These requests are hitting the LHR colo, latencies for R2 also seem to be fluctuating pretty badly for these same requests

Zack•7/29/24, 11:26 AM

Maybe it's just an issue ongoing at that particular colo? Requests from other parts of the world such as Toronto, Berlin, Kyiv are all responding fast

thomasgauvin•7/29/24, 9:42 PM

Thanks for the details @Zack that you shared in DMs, we'll keep our eyes open for LHR (UK region) latencies if it returns and investigate in our past logs.

Jay•7/30/24, 1:14 AM

I'm seeing some strange behavior around my KV right now, where for some reason deleted data is persisting.
I've attached two screenshots of what's going on (empty KV but log stating there is data there):
This is the snippet:

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

Is it recommended to re-use the KV storage in multiple workers? Or should be create multiple KV ? F

Similar Threads

Similar Threads

Similar Threads