Cloudflare Developers•14mo ago

Is anyone else seeing elevated error rates on KV puts over the last week or so?

Kkw another quick question, will put update remove the expiration if I don’t includ...

thomasgauvin•11/12/24, 8:18 PM

Put won't remove the expiration if you don't include it, but it will overwrite it if you provide a new expiration

Ccarlos-alberto Is anyone else seeing elevated error rates on KV puts over the last week or so?

thomasgauvin•11/12/24, 8:19 PM

Can you share you namespace id?

thomasgauvin•11/14/24, 3:12 PM

Are you using the REST API? That has an account wide rate limit, and we’d recommend catching your operations using the bulk operations endpoints of the REST API

Peps•11/14/24, 6:29 PM

Continuing from https://ptb.discord.com/channels/595317990191398933/779390076219686943/1306685718621323308

It appears my reported read latencies are higher than they're supposed to be. Most of the reads are hot, but still my p90 is about 230ms, which is much higher than I would expect.

Namespace ID: 21fd681cd0074337b08b56b98d1bd07c21fd681cd0074337b08b56b98d1bd07c

cc @thomasgauvin

Peps•11/14/24, 6:30 PM

(I personally don't actively work on this specific project that uses this namespace, but I can grab any info if needed)

PPeps Continuing from https://ptb.discord.com/channels/595317990191398933/779390076219...

thomasgauvin•11/15/24, 12:46 PM

What’s happening here is that they are regionally cached but not locally cached, so they get tagged as cache but I would hypothesize that your traffic is still pretty spread out and doesn’t regularly have >1 RPS to the same key within a Cloudflare region

Original message was deleted

thomasgauvin•11/15/24, 12:48 PM

If this is the case, you’re probably getting Cloudflare-wide API rate limits. https://developers.cloudflare.com/fundamentals/api/reference/limits/#:~:text=The%20global%20rate%20limit%20for,receiving%20a%20HTTP%20429%20response.

If you have an enterprise account, you could contact your account team to increase these

Cloudflare Docs

Limits | Cloudflare Fundamentals docs

The global rate limit for the Cloudflare API is 1200 requests per five minutes per user, and applies cumulatively regardless of whether the request is made via the dashboard, API key, or API token.

thomasgauvin•11/15/24, 12:49 PM

How are you using KV within your app/can you share more context about the use case? Trying to understand what’s the right solution here

Original message was deleted

thomasgauvin•11/15/24, 8:35 PM

every account has account-wide rate limits of 1200 requests per 5 minutes. That applies to using Workers KV via the REST API (which is what the cloudflarecloudflare npm package is calling).

We know that's limiting and we have ongoing efforts to revisit how rate limiting is applied, but this is months away (requires per product instance rate limiting).

I strongly recommend using the Workers KV binding that should be using from within your Workers https://developers.cloudflare.com/kv/api/read-key-value-pairs/. Is there a reason why you aren't using the binding currently?

thomasgauvin•11/15/24, 8:50 PM

KV update:

2024-11-14: We updated our REST API docs to add granular errors to bulk operations endpoints for writes and deletes. https://developers.cloudflare.com/kv/platform/changelog/#2024-11-14

The response body of those endpoints now contain unsuccessful_keysunsuccessful_keys and successful_key_countsuccessful_key_count in the

result

result

field. https://developers.cloudflare.com/api/operations/workers-kv-namespace-write-multiple-key-value-pairs#Responses

This gives you a way to know which of the KV pairs have been properly written/deleted in a bulk operation and enables you to retry only the unsuccessful keys in the event of an error.

(CC @Workers KV for those who subscribed)

Tthomasgauvin What’s happening here is that they are regionally cached but not locally cached,...

Peps•11/15/24, 9:21 PM

ah I see, makes sense. Thanks for the clarification!

Chaika•11/17/24, 9:50 PM

?crossposting -> d1-database

Flare•11/17/24, 9:50 PM

Please do not post your question in multiple channels/post it multiple times per the rules at ⁠#

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Xxiaoyu2er hi i have a question, is kv faster than d1, is there any official benchmark? i ...

thomasgauvin•11/18/24, 3:13 PM

Hi @xiaoyu2er, that comparison is outdated. Workers KV is the fastest way to retrieve data from a Worker, especially for high read throughput.

Workers KV makes use of caching across the Cloudflare network internally to make sure that highly requested items are very fast. That means that it's ideal for configuration data, non-relational data, A/B tests, user preferences, that requires very high read throughput (high requests per second) and that have infrequent reads. The caching means that your data is effectively global.

You'd use D1 for your general application data. It can handle lower throughput but provides consistency, which you may need if you want to immediately see writes. This is why it's recommended for general application data.

The big part that is missing from the benchmark above, in addition to it being outdated, is regionality. KV is replicated globally, whereas D1 is currently single-region.

Jak•11/22/24, 4:09 PM

Hi - Is there a difference in the below two calls? I'm getting two different results for

const value = await env.NAMESPACE.list({ prefix: "testing" });

vs

a call to

https://api.cloudflare.com/client/v4
/accounts/{account_id}/storage/kv/namespaces/{namespace_id}/keys/prefix=testing

https://api.cloudflare.com/client/v4
/accounts/{account_id}/storage/kv/namespaces/{namespace_id}/keys/prefix=testing

It seems that the default should be 1000 if not provided. My storage has around 200-300 items with the prefix but the first one would return an arbitrary number at times(e.g. 50, 93) while the rest call would always return all of them. I wonder if the first case could be suffering from

It is possible to have an empty array in keys, but still have more keys to fetch, because recently expired or deleted keys ↗ must be iterated through but will not be included in the returned keys.

It is possible to have an empty array in keys, but still have more keys to fetch, because recently expired or deleted keys ↗ must be iterated through but will not be included in the returned keys.

where the deleted ones are counted as part of the limit but the second one is not?

Tthomasgauvin Hi @xiaoyu2er, that comparison is outdated. Workers KV is the fastest way to ret...

thomasgauvin•11/22/24, 8:03 PM

Hi folks, we added a section to the Choose a data or storage productChoose a data or storage product documentation to help compare and contrast KV and D1, hope it helps! https://developers.cloudflare.com/workers/platform/storage-options/#workers-kv-vs-d1

Kimitri•11/23/24, 7:08 AM

Hi folks, I have a question, I came from traditional Node.js Express REST apis that caches with mysql database data with Redis, so I don't know if I understand KV well, but it's like Redis and is good to use it to cache database list results for performance improvement, or is D1 enough for this purpose? Or the purpose of KV is cache the entire response from a Worker?

1984 Ford Laser•11/24/24, 5:21 AM

I know it isn't many reads, but man when KV is doing hot only reads it is fast

QyloMc•11/24/24, 5:23 AM

Hlo

julius•11/24/24, 11:32 AM

hey there - has the behavior with KV changed recently? we are getting KV hits for deleted key-pairs

some context:

the key has a cacheTTL of 24h
we delete the key through the REST API
the key is not in the KV if i prefix search the key on the dashbaord
a console.log() after the read in the worker still finds the old row

so I'm thinking either the delete is just taking longer than usual to propagate, or maybe something's changed, cause iirc we got a miss here previously

Jjulius hey there - has the behavior with KV changed recently? we are getting KV hits fo...

Kimitri•11/24/24, 1:54 PM

how can I delete a key ? I just saw in the doc the method put and get

pat me•11/24/24, 5:57 PM

i have 2 questions, full disclosure: i don't understand back end infrastructure too well

1# is kv as secure as d1? like can i store emails as keys on it? is that a good idea?? or will ANYONE be able to retrieve that info? the docs seem to imply it's fairly secure i think

2# how quick are updates if i write into kv? will that happen under 30 mins and be reflected everywhere?

if info about my use case is needed, i want to store a list of digital item's urls or ids (which i will fetch with ajax on client side)
stored as "email (who purchased) : json (nested list of items purchased)"

originally i thought user sign in sign up would be neat but i don't want the responsibility of storing passwords or anything for something as simple as "hey did the user purchase this product fully?", specially because payments will be handled face to face, not digitally and i will have to input emails and purchase list by hand anyway. i think getting email from google sign in or something then fetching the key : value pair seems reasonable and simple to me

Ppat me i have 2 questions, full disclosure: i don't understand back end infrastructure ...

Walshy•11/24/24, 6:42 PM

No one can ever read another persons resources without explicitly being allowed by that person

So, no. No one can read your KV unless you explicitly create like an API for them to do so

It takes up to 60 seconds for new values to be reflected

WWalshy 1. No one can ever read another persons resources without explicitly being allow...

pat me•11/24/24, 6:54 PM

thanks!

Isaac McFadyen•11/25/24, 3:36 AM

By #1 the caveat is that KV values can be viewed on the dashboard, but as Walshy said, they'd need access to your Cloudflare account to do that.

IIsaac McFadyen By #1 the caveat is that KV values can be viewed on the dashboard, but as Walshy...

pat me•11/25/24, 6:54 AM

as long as it's not publicly available like static websites, that's all good yes thanks

1.1.1.1APP•11/27/24, 7:58 PM

DNS over Discord: GitHub

View the DNS over Discord source code on GitHub at https://dns-over-discord.v4.wtf/github

�

😈 Donkey 💫•11/28/24, 1:03 PM

is this too much for KV read ops?
data is a json 30KB

�

😈 Donkey 💫•11/28/24, 1:05 PM

testing after read: https://blog.cloudflare.com/faster-workers-kv/

this is string value of 1KB. That's a lots

1984 Ford Laser•11/28/24, 1:05 PM

Yeah that's abnormal

1984 Ford Laser•11/28/24, 1:05 PM

I was skeptical till you sent that second image lol

�

😈 Donkey 💫•11/28/24, 1:08 PM

and mixed-data
Never reach any lower than 100ms

1984 Ford Laser•11/28/24, 1:09 PM

Where's your nearest PoP?

�

😈 Donkey 💫•11/28/24, 1:10 PM

all requests (chart) come from around the world
My local dev is near Singapore (SIN)

11984 Ford Laser I know it isn't many reads, but man when KV is doing hot only reads it is *fast*

1984 Ford Laser•11/28/24, 1:11 PM

Okay cool so not localised, absolutely abnormal

�

😈 Donkey 💫•11/28/24, 1:31 PM

I admit that more a half are cold reads, but compare to … CDN cache read? That’s too much for KV

1984 Ford Laser•11/28/24, 1:31 PM

Even my cold reads have never been higher than 300 or so

�

😈 Donkey 💫•11/28/24, 1:35 PM

pat me•12/1/24, 9:22 AM

how do i set the expiry "since modified"? (for a key using put)

or after a certain amount of time has passed since the key was last modified.

https://developers.cloudflare.com/kv/api/write-key-value-pairs/

all i see is expiry at a fixed time since epoch and expiry in seconds from now

JJacob Correct, every time you write to the key and include the `expirationTtl`, it wil...

pat me•12/1/24, 9:28 AM

ohhhhhh i modify the ttl at write time dam, that's a simple way to handle it lol

Ppat me ohhhhhh i modify the ttl at write time dam, that's a simple way to handle it lol

thomasgauvin•12/2/24, 5:44 PM

Glad you found what you were looking for! created a PR to make it easier to find https://github.com/cloudflare/cloudflare-docs/pull/18505

GitHub

Update write-key-value-pairs.mdx by thomasgauvin · Pull Request #18...

Agent Beans•12/4/24, 4:53 PM

Does a KV get erased if its parent worker is redoployed? Is there a way to prevent that?

AAgent Beans Does a KV get erased if its parent worker is redoployed? Is there a way to preve...

Isaac McFadyen•12/4/24, 4:58 PM

It does not, no.

Isaac McFadyen•12/4/24, 4:59 PM

All data will stay in KV unless the namespace is deleted or the data is deleted by you.

IIsaac McFadyen It does not, no.

Agent Beans•12/4/24, 5:00 PM

Okay thanks - I dont think im deleting my files, but they had seemed to be going away on deployment - but ill keep investingating

Update: i was setting expirationTtl

Tthomasgauvin Glad you found what you were looking for! created a PR to make it easier to find...

pat me•12/5/24, 7:46 AM

thanks!

toadfans•12/13/24, 3:08 AM

Hello, is there a KV outage? I'm getting this error:

KV PUT failed: 500 Internal Server Error

KV PUT failed: 500 Internal Server Error

requestId:

8f12b7cdd8abcbee

8f12b7cdd8abcbee

accountId:

75d8d68ecf8f9e8155f1ccc7dad82e00

75d8d68ecf8f9e8155f1ccc7dad82e00

kv id

a9486de811d14c909ec398298d4d392e

a9486de811d14c909ec398298d4d392e

Is anyone else seeing elevated error rates on KV puts over the last week or so?

Similar Threads

Similar Threads

Similar Threads