I see, thanks for the info. Are durable objects really a viable replacement for KV? To be a bit more

SStoke I see, thanks for the info. Are durable objects really a viable replacement for ...

H

Hard@Work•7/29/24, 8:50 AM

No, they are only billable when processing requests, even if they are still "active"

S

StokeOP•7/29/24, 9:35 AM

Ok thanks. Every request activates for 30s, if I understand correctly. Makes sense. Then after reading through the docs I can see that the recommendationis to not use single global objects, so I think "object per link entity" would be the way to go here. I still struggle to wrap my head around why/how Durable Objects are the best fit for this use case. They're certainly cool and powerful, but I don't really need all the extra guarantees. All I'd really need is a bit of a narrower windows for eventual consistency in KV. I think not having a TTL and the first lookup being "a bit slow" are both real downsides for me, albeit both would be acceptable. I'm mainly worried about the complexity here... KV is very simple to grasp, whereas DO is quite complex conceptually, even when misused as a key-value store

Z

Zack•7/29/24, 11:22 AM

Hey, I am seeing some pretty horrendous latencies using workers with KV, requests to get a 3.6MB value are taking up to 10 seconds at times, this is from a london IP and the KV was created by me from an IP in the UK too. Not sure why sometimes the latencies are so bad, I was using this to speed up a bunch of queries that we would usually make to R2 as range requests, but it appears sometimes KV is substantially slower? Any ideas what could be going on here?

Z

Zack•7/29/24, 11:24 AM

These requests are hitting the LHR colo, latencies for R2 also seem to be fluctuating pretty badly for these same requests

Z

Zack•7/29/24, 11:26 AM

Maybe it's just an issue ongoing at that particular colo? Requests from other parts of the world such as Toronto, Berlin, Kyiv are all responding fast

T

thomasgauvin•7/29/24, 9:42 PM

Thanks for the details @Zack that you shared in DMs, we'll keep our eyes open for LHR (UK region) latencies if it returns and investigate in our past logs.

J

Jay•7/30/24, 1:14 AM

I'm seeing some strange behavior around my KV right now, where for some reason deleted data is persisting.
I've attached two screenshots of what's going on (empty KV but log stating there is data there):
This is the snippet:

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

const alreadyTracked = await env.<namespace>.get(<item>);
console.log('hit alreadyTracked', alreadyTracked);

J

Jay•7/30/24, 1:15 AM

Is this a bug?

env.<namespace>.list()

env.<namespace>.list()

env.<namespace>.list()

env.<namespace>.list() correctly shows nothing, but for this key it has persisted across multiple reloads, not to mention operations that used to work now just silently failing (

put

put

put

put, for one that I noticed)

J

Jay•7/30/24, 1:25 AM

I don't know if its some sort of race condition considering I'm doing multiple KV operations across different namespaces. Is there a "safe" way to put/delete data that I'm missing?

C

Chaika•7/30/24, 1:25 AM

KV's eventually consistent, both the dash and list go back to one of the central stores irrc. Are you overriding cache ttl or just using the default?

J

Jay•7/30/24, 1:25 AM

I don't have any cache set, so probably default

J

Jay•7/30/24, 1:26 AM

Have never seen a delay in consistency this long though

C

Chaika•7/30/24, 1:26 AM

default is just 60s

J

Jay•7/30/24, 1:26 AM

This has gone over 10 minutes

J

Jay•7/30/24, 1:26 AM

still happening with this key

J

Jay•7/30/24, 1:27 AM

same log new timestamp to show ^

J

Jay•7/30/24, 1:30 AM

The value isn't overly large or anything, its just a stringified object about as long as this message

C

Chaika•7/30/24, 1:31 AM

yea, weird. I don't see any issues, just updated and deleted a key to test. You said PUTs were failing as well, to the same key?

J

Jay•7/30/24, 1:33 AM

Yeah so I had a line that looked like this to test (and I'm aware I believe just putting overwrites but I was testing something iirc):

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

await env.<namespace>.delete(<item>)
await env.<namespace>.put(<item>)

which:

showed the key as deleted in the CF dashboard
did not run the
```
put
```
put
```
put
```
put
according to the dashboard
key is still retrievable

J

Jay•7/30/24, 1:34 AM

Let me check if the put went through just programmatically

J

Jay•7/30/24, 1:35 AM

Ok yeah the put went through programmatically its just not updating on the CF dash

J

Jay•7/30/24, 1:37 AM

agh so it seems like things are fine internally the dashboard is just never to be trusted

A

Anthony•8/2/24, 9:43 AM

How do you modify every KV entry without being ratelimited?

"Error: Too many API requests by single worker invocation."

B

Ben•8/2/24, 5:21 PM

Ok so I have a question regarding the KV list and specifically the cursor.

I want to list all values of a workers KV store. I am using a cursor to "go through" the entire KV store. What happens if a new entry is created with a brand new key? Would I need to list through the KV store again or is it guaranteed to exist at the end of my pagination? Thanks!

J

jason•8/2/24, 7:22 PM

I'm using Cloudflare Pages with KV.

My goal is to set up KV for different environments, to have:

one KV id for production, and
another KV id for all other environments (preview URLs, local dev with miniflare maybe, etc)

The docs show two approaches:

v1

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

kv_namespaces = [
  { binding = "KV", id = "abc123", preview_id = "xyz456" }
]

OR

v2 (using environments)

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

kv_namespaces = [
  { binding = "MY_KV", id = "abc123" }
]

[env.production]
route = "example.com/*"
kv_namespaces = [
  { binding = "MY_KV", id = "xyz456" }
]

which is preferred and why?
for the first version, how does Cloudflare differentiate when to use the KV represented by
```
id
```
id
```
id
```
id
or
```
preview_id
```
preview_id
```
preview_id
```
preview_id
?

K

kian•8/2/24, 7:23 PM

preview_id

preview_id

preview_id

preview_id is related to

wrangler dev

wrangler dev

wrangler dev

wrangler dev, not environments

J

jason•8/2/24, 7:24 PM

say more please

A

Abakir•8/7/24, 5:37 PM

what is the purpose of metadata for key value pairs cloudflare? Why would you use that verses just using a value?

AAbakir what is the purpose of metadata for key value pairs cloudflare? Why would you us...

I

Isaac McFadyen•8/7/24, 5:38 PM

Metadata can be retrieved using a

list

list

list

list call, values cannot.

I

Isaac McFadyen•8/7/24, 5:39 PM

So if you have a couple thousand keys, and store important data in the metadata, you can get all that data with one call versus iterating through and using 1000+ calls (and being billed for 1000+ calls)

IIsaac McFadyen So if you have a couple thousand keys, and store important data in the metadata,...

A

Abakir•8/7/24, 5:40 PM

This makes sense thank you

J

jacob•8/7/24, 9:55 PM

Anybody who's managed to use KV with container based DI like tsyringe? I find that the kv namespace methods break once put through tsyringe. Specifically in classes using it as a dependency. I get super weird behaviour where the methods don't throw any sort of error but seemingly just stop any execution of code coming after it.

J

jacob•8/7/24, 10:27 PM

It actually appears to be unrelated to the container. I'm seeing the same behaviour simply passing it down a constructor, the kv namespace methods become unusable.

J

jacob•8/7/24, 10:45 PM

Here's an example of passing the kv down (token manager is effectively a wrapper of my particular kv namespace)

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

import { TokenManager, TokenType } from "@/managers/TokenManager";

export class VisionService {

  constructor(private tokenManager: TokenManager) { }

  async annotate() {
    console.log('hello world 1')
    const token = await this.tokenManager.get(TokenType.GOOGLE_CLOUD_VISION);
    console.log(token);
    console.log('hello world 2')
  }

}

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

  const tokenManager = new TokenManager(c.env);

  // Logs token from kv as expected
  console.log(await tokenManager.get(TokenType.GOOGLE_CLOUD_VISION));

  const visionService = new VisionService(tokenManager);

  // Does not log token
  visionService.annotate()

Console only outputs ..

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

<token>
hello world 1

expected

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

<token>
hello world 1
<token>
hello world 2

D

Darrell•8/8/24, 11:27 AM

Has expiration_ttl now gone from the V4 API (for a KV write with metadata). I can only see it in the OpenAPI spec comment and it doesn't seem to work in practice anymore.

Original message was deleted

J

jacob•8/8/24, 5:03 PM

Oh my god :NotLikeThis: thank you so much!

00xJM Were there any adjustments to how Cloudflare processed KV updates from the API l...

C

Chaika•8/9/24, 3:20 PM

https://www.cloudflarestatus.com/incidents/4lw1rrpc0fhz

Workers KV REST API Bulk Upload issues

M

mihaaai•8/13/24, 9:15 PM

Bug on the KV docs for pricing.
Since you changed the docs layout I get this misleading pricing for KV, it was 5$/million if I remember correctly, now shows

.00/million

.00/million

.00/million

.00/million
Coming from the server, it's not a front-end glitch, please fix it, or face the lawsuits

C

Chaika•8/13/24, 9:24 PM

lol thankfully the dashboard one /subscriptions still shows the right cost, should def be fixed though, thanks for reporting

Mmihaaai Bug on the KV docs for pricing. Since you changed the docs layout I get this mis...

K

kian•8/13/24, 9:38 PM

Fix is on the way, thanks for reporting it!

Mmihaaai Bug on the KV docs for pricing. Since you changed the docs layout I get this mis...

K

Kody•8/13/24, 10:11 PM

Fix is merged, should be live in ~10 min.

Thanks so much for reporting

Z

Zack•8/15/24, 9:08 PM

Can someone give some idea of what the rate limit is for writing to KV using the API? Trying to pre-populate the KV store with around 2.8k entries prior to deployment, but hitting rate limits.

Z

Zack•8/15/24, 9:08 PM

Would using the bulk API avoid this?

Z

Zack•8/15/24, 9:09 PM

I was using the individual API because I wanted to stream the body to fetch

Z

Zack•8/16/24, 10:39 AM

The above question I think I found the answer as API's appear to be limited to 1200 requests/5 minutes for basically any API on cloudflare.

Z

Zack•8/16/24, 10:41 AM

I do have another question though about the architecture of KV. I know that KV gets are cached in the local colo CDN, but I was wondering where cold requests come from. Is KV single-homed per namespace, or is the KV data distributed to multiple locations globally to reduce latency for cold hits?

Z

Zack•8/16/24, 10:41 AM

This seems to indicate that it is multi-homed where it says "Central data stores", just wondering if this is accurate and where exactly those "homes" are?

ZZack This seems to indicate that it is multi-homed where it says "Central data stores...

H

Hard@Work•8/16/24, 12:25 PM

EU and US are the two central stores

HHard@Work EU and US are the two central stores

Z

Zack•8/16/24, 12:25 PM

I see, are there any plans to expand this?

I see, thanks for the info. Are durable objects really a viable replacement for KV? To be a bit more

Similar Threads

Similar Threads

Similar Threads