Thanks both, I've passed on all the info to the KV engineering. Hopefully someone with more domain k

Thanks both, I've passed on all the info to the KV engineering. Hopefully someone with more domain knowledge from CF will pick this up when they see this

JJun Thanks both, I've passed on all the info to the KV engineering. Hopefully someon...

ivogt•10/23/24, 12:20 PM

Just to confirm that it doesn't look like isulated case we are seeing this on multiple deployments across regions.

thomasgauvin•10/23/24, 12:32 PM

Thanks everyone, we're looking into this

thomasgauvin•10/23/24, 1:04 PM

@danielsundq @ivogt @Anthony can you share in this thread since when you've been encountering these issues?

ivogt•10/23/24, 1:10 PM

I believe it started after 8:00 UTC yesterday. Speculating on the timing but definately wasn't there the day before. Also we haven't had such problems for months.

ivogt•10/23/24, 1:12 PM

Possibley somethig that happend in the afternoon as we started noticing past 12:00 UTC .

danielsundq•10/23/24, 1:24 PM

We detected the problem this morning, about 06:00 UTC. I don't think it was a problem the day before, because this latency causes big issues with our applications and we didn't notice anything the day before

thomasgauvin•10/23/24, 1:27 PM

Can you all DM me your Workers account id and KV namespace id

LuKks•10/23/24, 7:21 PM

In case someone uses Workers KV in Node.js, feedback welcomed

https://github.com/LuKks/workers-kv

GitHub

GitHub - LuKks/workers-kv: Eventual-consistent key-value data stora...

Eventual-consistent key-value data storage based on Workers KV - LuKks/workers-kv

Advany•10/24/24, 6:27 PM

I just read: https://blog.cloudflare.com/how-we-built-cloudflare-queues/

Also the part "Workers KV acts as a fast metadata store: our Worker can quickly look up the shard map to perform load balancing across shards"

How can they use KV for that, as KV has a 60 second delay to update after writing? Won't the entire infrastructure have a 60 second delay to understand what the state is and how would the be able to update that state?

The Cloudflare Blog

Durable Objects aren't just durable, they're fast: a 10x speedup fo...

Learn how we built Cloudflare Queues using our own Developer Platform and how it evolved to a geographically-distributed, horizontally-scalable architecture built on Durable Objects. Our new architecture supports over 10x more throughput and over 3x lower latency compared to the previous version.

Paco ・ francisco.io•10/29/24, 12:06 PM

I wrote a Cloudflare KV wrapper! I'd love feedback and if you'd add/remove things, including things that are not clear or could be improved (to the library overall or the KV workers bit). I almost did it with the REST API (to avoid having to init it within the

fetch()

fetch()

context and not at the root). But decided the fetch context (only the namespace_idnamespace_id needed + location within that context) might be more worth it than having to force the users to find their account_idaccount_id , the namespace_idnamespace_id and the Bearer tokenBearer token . Would've preferred a lot more ofc if it was available at the root, but ALAS. Here is the Cloudflare KV-specific docs!
https://polystore.dev/documentation#cloudflare-kv

🏬 Polystore - A universal library for standardizing any KV-store

Add a unified API for any KV store like localStorage, Redis, FileSystem, etc.

good_for_nothing•10/29/24, 2:38 PM

Hey, just a general question, is KV appropriate for creating a authorization layer in my application created using worker?

I think we are okay with the eventual consistency of KV as long as it's under a minute or two.

The KV will be responsible for storing the data and we will create our own logic.

I am pretty much a noob when it comes to AuthZ, so is there a better solution with cloudflare worker? I am okay with a SaaS or an open source solution if available. Thank you.

Ggood_for_nothing Hey, just a general question, is KV appropriate for creating a authorization lay...

Ben•10/29/24, 3:22 PM

Any reason why you wouldn’t use d1/serverless database?

Ussssuuaaallllyyyyyy KV stores are meant to be for value caches rather than persistent stores of data.
As in, if you wiped the KV cache, your application would just fill it up again next time around.

(Note: I’m wicked new to the Cloudflare hosted services, but 15 years of other services)

good_for_nothing•10/29/24, 3:38 PM

I hadnt looked into it. The reason KV caught my eyes was because of its low latency and similarities to redis. Thank you for the suggestion

thomasgauvin•10/29/24, 3:42 PM

@good_for_nothing KV is persistent storage, using KV as an authorization layer is definitely a good use case, lots of folks use KV for storing nosql data such as allow/deny lists, auth lists, a/b testing groups, other configuration.

Tthomasgauvin @good_for_nothing KV is persistent storage, using KV as an authorization layer i...

Stan Angeloff•10/29/24, 3:50 PM

I recently re-read the "How KV works" article and wanted to flag this up from the article:

KV is not ideal for applications where you need support for atomic operations or where values must be read and written in a single transaction.

In that regard I agree authorization is a good candidate, however lots of developers likely get it confused with authentication and session management — KV might not be a good case for sessions due to it being eventually-consistent. Thoughts I'd bring this up since I have had people on the team ask similar questions.

thomasgauvin•10/29/24, 3:52 PM

That's correct, if you need consistent access to your data today, you're better off using another service. However, @good_for_nothing noted I think we are okay with the eventual consistency of KV as long as it's under a minute or two.I think we are okay with the eventual consistency of KV as long as it's under a minute or two. and you will definitely attain that with Workers KV.

thomasgauvin•10/29/24, 4:39 PM

On this topic, we're actually currently looking to provide consistent reads in Workers KV to give you more flexibility in how you use KV, if you're interested, please let us know your use case in this thread or DM me!

Tthomasgauvin On this topic, we're actually currently looking to provide consistent reads in W...

Stan Angeloff•10/29/24, 6:34 PM

So KV will turn more like, um, Redis server rather than a Redis cluster, with strong atomic guarantees?

Avi•10/30/24, 4:23 AM

looking for advice - i'm hoping to build a dynamic redirector for an internal tool.

so, staging.example.com -> pr-123.example.workers.dev

Avi•10/30/24, 4:24 AM

what might be a good way to implement this? i'm thinking of a small worker that acts like a transparent proxy, and the proxy URL would be stored in a KV value

Avi•10/30/24, 4:24 AM

the idea is we'd be able to dynamically change the redirect on the fly, like /preview 123

Avi•10/30/24, 4:24 AM

and it'd work ~immediately after

Avi•10/30/24, 4:27 AM

i guess another alternative would be creating a Single Redirect rule via the cloudfalre REST API

AAvi looking for advice - i'm hoping to build a dynamic redirector for an internal to...

Ben•10/30/24, 10:04 AM

Why not just modify dns to CNAME to the worker of your choice?

BBen Why not just modify dns to CNAME to the worker of your choice?

Hello, I’m Allie!•10/30/24, 1:38 PM

Assuming that both are on CF, then you can’t just CNAME them

PPaco ・ francisco.io I wrote a Cloudflare KV wrapper! I'd love feedback and if you'd add/remove thing...

DiamondDragon•10/30/24, 2:59 PM

Amazing- I was writing my own kv methods with the API as I didn’t find another library out there to use kv outside of a worker

BBen Why not just modify dns to CNAME to the worker of your choice?

Avi•10/30/24, 4:32 PM

yes that’s right, you can’t CNAME to another worker afaik

Avi•10/30/24, 4:32 PM

i ended up just building a worker to act as a transparent proxy

AAvi looking for advice - i'm hoping to build a dynamic redirector for an internal to...

thomasgauvin•10/31/24, 4:49 PM

This seems very similar to a URL shortener. Very valid, KV is a good choice

thomasgauvin•11/4/24, 7:26 PM

KV update:

2024-11-04: We recently updated our REST API docs to reflect that Workers KV analytics REST APIs (https://developers.cloudflare.com/api/operations/workers-kv-request-analytics-query-request-analytics and https://developers.cloudflare.com/api/operations/workers-kv-stored-data-analytics-query-stored-data-analytics) are deprecated and will be shut down on January 31, 2025.

This follows our previous changelog entry of this deprecation and announcement of the Workers KV GraphQL API https://developers.cloudflare.com/kv/platform/changelog/#2024-08-08. The GraphQL API is the way to get analytics for your Workers KV namespaces going forward, since it provides a lot of the granularity and flexibility that you've been asking for such as per-namespace analytics and is unified with the rest of the developer platform products.

If you are using the Workers KV analytics REST APIs (https://developers.cloudflare.com/api/operations/workers-kv-request-analytics-query-request-analytics and https://developers.cloudflare.com/api/operations/workers-kv-stored-data-analytics-query-stored-data-analytics), migrate to the new GraphQL API for Workers KV analytics. Refer to our docs for more information on the GraphQL API and sample queries https://developers.cloudflare.com/kv/observability/metrics-analytics/

You may have received an email recently if we've detected you're still using these KV analytics APIs reminding you of this deprecation. (CC @Workers KV for those who subscribed)

PJP•11/6/24, 4:17 PM

Does anyone have a reference code on how to create a full export of a kv namespace? I am trying to write one myself but I am running into a lot of different issues, ranging from invalid cursors to max subrequest errors and/or inconsistent results. The kv I am testing with shows about 8.5k records in the cloudflare ui where I am already running into issues, the different production kv I will need to export will have around 144k entries so I am a bit worried about being able to export it successfully.

My use-case is that we have a kv setup with redirects (key is the original url, target url is the value of the kv record), with data imported overtime from multiple files and we now sadly lack a "master file" with all our data. So to create this masterfile I want to export the current state of the kv into a csv so that the data can be analysed by business users in excel.

Mitya•11/7/24, 3:43 PM

Hi everyone, I'm new to KV and have been reading this guide. It talks about storing and serving static assets. My question is: what are the advantages of using KV for this rather than a public R2 bucket?

MMitya Hi everyone, I'm new to KV and have been reading [this guide](https://developers...

Hello, I’m Allie!•11/7/24, 4:11 PM

Well, this guide looks to be from back when Workers Sites were still a thing. You can use Assets now for free asset serving

Hello, I’m Allie!•11/7/24, 4:11 PM

Another thing too is that per-key RPS can scale a bit better with KV as compared to R2 + built-in caching

PPJP Does anyone have a reference code on how to create a full export of a kv namespa...

thomasgauvin•11/7/24, 5:23 PM

Like HardlyWorkin' said, there are a lot of better options than a public R2 bucket for serving static assets. Pages (https://developers.cloudflare.com/pages/) and Workers assets (https://developers.cloudflare.com/workers/static-assets/) provide hosting built for serving static sites. Using R2 and KV, while possible, would be sort of rebuilding what Pages & Workers assets do (can be valid in some cases!)

Tthomasgauvin Like HardlyWorkin' said, there are a lot of better options than a public R2 buck...

Hello, I’m Allie!•11/7/24, 5:32 PM

Actually, would be pretty cool to have Assets+, that could fallback to R2 instead of KV for large assets

Himakar•11/8/24, 6:47 AM

What is the maximum limit of kv namespace for enterprise edition?

HHimakar What is the maximum limit of kv namespace for enterprise edition?

fry69•11/8/24, 7:07 AM

These are the official limits -> https://developers.cloudflare.com/kv/platform/limits/
Otherwise talk to your enterprise support contact you should have when you are on an enterprise plan.

Cloudflare Docs

Limits | Cloudflare Workers KV

Tthomasgauvin Like HardlyWorkin' said, there are a lot of better options than a public R2 buck...

Mitya•11/8/24, 4:08 PM

Thanks, both.

kw•11/11/24, 8:49 PM

Error: KV PUT failed: 429 Too Many Requests
what is the rate limit on put?

Walshy•11/11/24, 10:19 PM

https://developers.cloudflare.com/kv/platform/limits/

Walshy•11/11/24, 10:19 PM

1 write per key per second

kw•11/12/24, 5:38 AM

another quick question, will put update remove the expiration if I don’t include it?

kv.put(key, value, {expiration: x}) // Create the KV pairs

kv.put(key,value2) // Update the value only

kw•11/12/24, 5:39 AM

In this case, is the expiration updated?

carlos-alberto•11/12/24, 4:16 PM

Is anyone else seeing elevated error rates on KV puts over the last week or so?

Kkw another quick question, will put update remove the expiration if I don’t includ...

thomasgauvin•11/12/24, 8:18 PM

Put won't remove the expiration if you don't include it, but it will overwrite it if you provide a new expiration

Ccarlos-alberto Is anyone else seeing elevated error rates on KV puts over the last week or so?

thomasgauvin•11/12/24, 8:19 PM

Can you share you namespace id?

thomasgauvin•11/14/24, 3:12 PM

Are you using the REST API? That has an account wide rate limit, and we’d recommend catching your operations using the bulk operations endpoints of the REST API

Peps•11/14/24, 6:29 PM

Continuing from https://ptb.discord.com/channels/595317990191398933/779390076219686943/1306685718621323308

It appears my reported read latencies are higher than they're supposed to be. Most of the reads are hot, but still my p90 is about 230ms, which is much higher than I would expect.

Namespace ID:

21fd681cd0074337b08b56b98d1bd07c

21fd681cd0074337b08b56b98d1bd07c

cc @thomasgauvin

Thanks both, I've passed on all the info to the KV engineering. Hopefully someone with more domain k

Similar Threads

Similar Threads

Similar Threads