When you write to KV, the changes are immediately made to KV's origin stores. They are also immediat

M

Marcus•7/27/24, 6:19 PM

Is it (still?) true that there's no way to move a KV namespace to another Cloudflare account besides writing a script to bulk-download every key/value pair from one and bulk-upload to another?

M

Marcus•7/27/24, 7:03 PM

(and that would consume 1 write per key, so on the free tier this would have to be spread over ~

keys/1000

keys/1000

keys/1000

keys/1000 days?)

M

Marcus•7/27/24, 7:26 PM

I see a bulk write endpoint, but it only takes JSON and our values are binary so that doesn't help

W

Walshy•7/27/24, 8:18 PM

you can provide base64

W

Walshy•7/27/24, 8:19 PM

https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-multiple-key-value-pairs

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

W

Walshy•7/27/24, 8:19 PM

base64
boolean
Whether or not the server should base64 decode the value before storing it. Useful for writing values that wouldn't otherwise be valid JSON strings, such as images.

M

Marcus•7/27/24, 9:07 PM

ooh, interesting. I guess I could download one-by-one, convert to base64 locally, bulk upload. do you know if that endpoint counts as a single write re: the daily limits?

W

Walshy•7/27/24, 9:11 PM

https://developers.cloudflare.com/kv/platform/pricing/#frequently-asked-questions

Each key-value pair in the (bulk write) PUT request is counted as a single write, identical to how each call to PUT in the Workers API counts as a write. Writing 5,000 keys via the REST API incurs the same write costs as making 5,000 PUT calls in a Worker.

J

John Spurlock•7/28/24, 10:35 PM

nice! new KV storage and operations datasets available in the cf graphql api:

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups

kvOperationsAdaptiveGroups and

kvStorageAdaptiveGroups

kvStorageAdaptiveGroups

kvStorageAdaptiveGroups

kvStorageAdaptiveGroups

was always a bit strange having to go the rest api for those

MMarcus ooh, interesting. I guess I could download one-by-one, convert to base64 locally...

M

Marcus•7/28/24, 11:05 PM

resulting (somewhat hacky) script, in case someone else needs to do this: https://github.com/marcustyphoon/cloudflare-kv-migrate

Tthomasgauvin When you write to KV, the changes are immediately made to KV's origin stores. Th...

S

Stoke•7/29/24, 8:46 AM

I see, thanks for the info. Are durable objects really a viable replacement for KV? To be a bit more concrete on the use case; let's imagine a link shortener service with 100k links (per month) and 20M clicks per month. My real/actual use case is close enough to this example scenario. I would store link id -> link URL as key -> value. The load is very read heavy with spikes of over 200 rps, but there are only very few updates to existing keys, let's say 1 per hour. I would traditionally use Redis (or similar) for such a use case, that's why I'm looking at KV. Using durable objects, this would be a single object for the entire map of links? Or one object per link (i.e. 100k objects per month)? Since duration is billed, would the latter mean that 100k objects per month are "active" indefinitely?

SStoke I see, thanks for the info. Are durable objects really a viable replacement for ...

H

Hard@Work•7/29/24, 8:50 AM

No, they are only billable when processing requests, even if they are still "active"

S

Stoke•7/29/24, 9:35 AM

Ok thanks. Every request activates for 30s, if I understand correctly. Makes sense. Then after reading through the docs I can see that the recommendationis to not use single global objects, so I think "object per link entity" would be the way to go here. I still struggle to wrap my head around why/how Durable Objects are the best fit for this use case. They're certainly cool and powerful, but I don't really need all the extra guarantees. All I'd really need is a bit of a narrower windows for eventual consistency in KV. I think not having a TTL and the first lookup being "a bit slow" are both real downsides for me, albeit both would be acceptable. I'm mainly worried about the complexity here... KV is very simple to grasp, whereas DO is quite complex conceptually, even when misused as a key-value store

Z

Zack•7/29/24, 11:22 AM

Hey, I am seeing some pretty horrendous latencies using workers with KV, requests to get a 3.6MB value are taking up to 10 seconds at times, this is from a london IP and the KV was created by me from an IP in the UK too. Not sure why sometimes the latencies are so bad, I was using this to speed up a bunch of queries that we would usually make to R2 as range requests, but it appears sometimes KV is substantially slower? Any ideas what could be going on here?

Z

Zack•7/29/24, 11:24 AM

These requests are hitting the LHR colo, latencies for R2 also seem to be fluctuating pretty badly for these same requests

Z

Zack•7/29/24, 11:26 AM

Maybe it's just an issue ongoing at that particular colo? Requests from other parts of the world such as Toronto, Berlin, Kyiv are all responding fast

T

thomasgauvinOP•7/29/24, 9:42 PM

Thanks for the details @Zack that you shared in DMs, we'll keep our eyes open for LHR (UK region) latencies if it returns and investigate in our past logs.

J

Jay•7/30/24, 1:14 AM

I'm seeing some strange behavior around my KV right now, where for some reason deleted data is persisting.
I've attached two screenshots of what's going on (empty KV but log stating there is data there):
This is the snippet:

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

J

Jay•7/30/24, 1:15 AM

Is this a bug?

env.<namespace>.list()

env.<namespace>.list()

env.<namespace>.list()

env.<namespace>.list() correctly shows nothing, but for this key it has persisted across multiple reloads, not to mention operations that used to work now just silently failing (

put

put

, for one that I noticed)

J

Jay•7/30/24, 1:25 AM

I don't know if its some sort of race condition considering I'm doing multiple KV operations across different namespaces. Is there a "safe" way to put/delete data that I'm missing?

C

Chaika•7/30/24, 1:25 AM

KV's eventually consistent, both the dash and list go back to one of the central stores irrc. Are you overriding cache ttl or just using the default?

J

Jay•7/30/24, 1:25 AM

I don't have any cache set, so probably default

J

Jay•7/30/24, 1:26 AM

Have never seen a delay in consistency this long though

C

Chaika•7/30/24, 1:26 AM

default is just 60s

J

Jay•7/30/24, 1:26 AM

This has gone over 10 minutes

J

Jay•7/30/24, 1:26 AM

still happening with this key

J

Jay•7/30/24, 1:27 AM

same log new timestamp to show ^

J

Jay•7/30/24, 1:30 AM

The value isn't overly large or anything, its just a stringified object about as long as this message

C

Chaika•7/30/24, 1:31 AM

yea, weird. I don't see any issues, just updated and deleted a key to test. You said PUTs were failing as well, to the same key?

J

Jay•7/30/24, 1:33 AM

Yeah so I had a line that looked like this to test (and I'm aware I believe just putting overwrites but I was testing something iirc):

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

which:

showed the key as deleted in the CF dashboard
did not run the
```
put
```
```
put
```
according to the dashboard
key is still retrievable

J

Jay•7/30/24, 1:34 AM

Let me check if the put went through just programmatically

J

Jay•7/30/24, 1:35 AM

Ok yeah the put went through programmatically its just not updating on the CF dash

J

Jay•7/30/24, 1:37 AM

agh so it seems like things are fine internally the dashboard is just never to be trusted

A

Anthony•8/2/24, 9:43 AM

How do you modify every KV entry without being ratelimited?

"Error: Too many API requests by single worker invocation."

B

Ben•8/2/24, 5:21 PM

Ok so I have a question regarding the KV list and specifically the cursor.

I want to list all values of a workers KV store. I am using a cursor to "go through" the entire KV store. What happens if a new entry is created with a brand new key? Would I need to list through the KV store again or is it guaranteed to exist at the end of my pagination? Thanks!

J

jason•8/2/24, 7:22 PM

I'm using Cloudflare Pages with KV.

My goal is to set up KV for different environments, to have:

one KV id for production, and
another KV id for all other environments (preview URLs, local dev with miniflare maybe, etc)

The docs show two approaches:

v1

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

OR

v2 (using environments)

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

which is preferred and why?
for the first version, how does Cloudflare differentiate when to use the KV represented by
```
id
```
```
id
```
or
```
preview_id
```
```
preview_id
```
?

K

kian•8/2/24, 7:23 PM

preview_id

preview_id

is related to

wrangler dev

wrangler dev

, not environments

J

jason•8/2/24, 7:24 PM

say more please

A

Abakir•8/7/24, 5:37 PM

what is the purpose of metadata for key value pairs cloudflare? Why would you use that verses just using a value?

AAbakir what is the purpose of metadata for key value pairs cloudflare? Why would you us...

I

Isaac McFadyen•8/7/24, 5:38 PM

Metadata can be retrieved using a

list

list

call, values cannot.

I

Isaac McFadyen•8/7/24, 5:39 PM

So if you have a couple thousand keys, and store important data in the metadata, you can get all that data with one call versus iterating through and using 1000+ calls (and being billed for 1000+ calls)

IIsaac McFadyen So if you have a couple thousand keys, and store important data in the metadata,...

A

Abakir•8/7/24, 5:40 PM

This makes sense thank you

J

jacob•8/7/24, 9:55 PM

Anybody who's managed to use KV with container based DI like tsyringe? I find that the kv namespace methods break once put through tsyringe. Specifically in classes using it as a dependency. I get super weird behaviour where the methods don't throw any sort of error but seemingly just stop any execution of code coming after it.

J

jacob•8/7/24, 10:27 PM

It actually appears to be unrelated to the container. I'm seeing the same behaviour simply passing it down a constructor, the kv namespace methods become unusable.

J

jacob•8/7/24, 10:45 PM

Here's an example of passing the kv down (token manager is effectively a wrapper of my particular kv namespace)

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

Console only outputs ..

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

expected

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

D

Darrell•8/8/24, 11:27 AM

Has expiration_ttl now gone from the V4 API (for a KV write with metadata). I can only see it in the OpenAPI spec comment and it doesn't seem to work in practice anymore.

Original message was deleted

J

jacob•8/8/24, 5:03 PM

Oh my god :NotLikeThis: thank you so much!

00xJM Were there any adjustments to how Cloudflare processed KV updates from the API l...

C

Chaika•8/9/24, 3:20 PM

https://www.cloudflarestatus.com/incidents/4lw1rrpc0fhz

Workers KV REST API Bulk Upload issues

M

mihaaai•8/13/24, 9:15 PM

Bug on the KV docs for pricing.
Since you changed the docs layout I get this misleading pricing for KV, it was 5$/million if I remember correctly, now shows

.00/million

.00/million

.00/million

.00/million
Coming from the server, it's not a front-end glitch, please fix it, or face the lawsuits

When you write to KV, the changes are immediately made to KV's origin stores. They are also immediat

Similar Threads

Similar Threads

Similar Threads