I think the issue is that a lot of backend config stuff is stored in KV in US Central, which also ha

I think the issue is that a lot of backend config stuff is stored in KV in US Central, which also happened to be the location hardest hit by the looks of it

parisholley•6/12/25, 8:08 PM

didn't last big outage show holes in their config architecture? thought that was all being mitigated

Sam•6/12/25, 8:09 PM

Error listing KV entries: KV GET failed: 503 Service temporarily unavailableError listing KV entries: KV GET failed: 503 Service temporarily unavailable Hopefully it hits me soon

Sam•6/12/25, 8:10 PM

ayyy there it is

11984 Ford Laser I think the issue is that a lot of backend config stuff is stored in KV in US Ce...

Chaika•6/12/25, 8:12 PM

important backend stuff/anything config should be QS (Quicksilver, Cf's own replicated edge kv db)

Chaika•6/12/25, 8:12 PM

that's why websites/cdn/proxy continued to work fine

Chaika•6/12/25, 8:13 PM

KV is (clearly now) GCP dependent, and CF Access, dashboard, etc, depend onKV

ddsgadget•6/12/25, 8:16 PM

all my kv are still down, i guess they only recover KV by geographic location or something?

Dddsgadget all my kv are still down, i guess they only recover KV by geographic location or...

Chaika•6/12/25, 8:18 PM

here's def still some errors, thundering herd problem and all that, recovery is slow

Chaika•6/12/25, 9:04 PM

All Cloudflare services have been restored and are now fully operational. We are moving the incident to Monitoring while we watch platform metrics to confirm sustained stability.

Reikoshea•6/12/25, 9:12 PM

all my sites use KV, and they're all back but 1. For some reason that 1 is still throwing the KV 503.

Reikoshea•6/12/25, 9:13 PM

throws it on both the site and the in the worker ide.

ddsgadget•6/12/25, 9:18 PM

for me it's starting to come back. One thing to say about Cloudflare is that they "eat their own cooking", so if something like KV goes down, even they are screwed, and they've got a lot to lose. So in that sense, issues with something like KV are attended to almost immediately. That being said, this was a pretty bad outage and for me at least, I am reluctant to rely on it as much as I have

Reikoshea•6/12/25, 9:19 PM

The same is true for pretty much everyone. When kenisis went down in AWS a couple years ago, it took out most of AWS's provisioning and updating tools

ddsgadget•6/13/25, 1:32 AM

Just started getting KV failures again. This thing is not stable.

1984 Ford LaserOP•6/13/25, 2:16 AM

Seems to be working fine for me

AZipr•6/13/25, 4:07 AM

It was a major outage. CF GCP and AWS were affected.

https://www.livemint.com/news/us-news/netizens-claim-major-internet-outage-caused-by-cyberattack-all-details-on-cloudflare-gcp-downtime-here-11749765997009.html#

Chaika•6/13/25, 4:14 AM

nah, aws was fine. It was just GCP failing which cascaded into CF/Spotify/etc (and CF's customers, etc)

AAZipr It was a major outage. CF GCP and AWS were affected. https://www.livemint.com...

crossbeau•6/13/25, 4:34 AM

Author is drunk. They wrote wednesday many times

AZipr•6/13/25, 5:07 AM

:bibicat:

thomasgauvin•6/13/25, 2:40 PM

Hi folks, we published an update on the service outage yesterday here: https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/

MMallissin I was reading the post mortem and I'm really afraid to ask this, but was Cloudfl...

Chaika•6/13/25, 6:28 PM

lol def not Google Sheets. They were using one of Google Cloud's products. They weren't specific on which ones they are dependent on, but you could make an educated guess

Chaika•6/13/25, 6:30 PM

Why are all the explanations being so vague?

Transparency is nice and it'd be cool to see a blog post about what exactly is used under the hood, but I mean for an incident report the bottom line is just: Cloudflare's products failed for hours due to an outage. Themselves, upstream or not, it's their responsibility to make systems reliable and not have downtime, and they detailed they're working towards that

imo you can't (and they didn't, they didn't even name GCP) just blame partners for outages, when you're the one responsible for picking them and ultimately providing the service.

MMallissin I was teasing and yes, can guess which particular service was used, but I still ...

Chaika•6/13/25, 10:22 PM

I would note that they said at one point they had another fail-over/partner

Workers KV removed a storage provider as we worked to re-architect KV’s backend

Moving the object store inhouse does not solve the issue of lack of fall-over. What if a problem with R2 causes a similar issue in the future?

I don't think they were really clear fully on their plans, maybe they plan on their own + GCP as a backup. R2 is their service/internal to them, even if they planned on going with just R2 if there's any issues it's internal to them/for them to fix. Basically them being in control of more the infra in the same way GCS does for example

Chaika•6/13/25, 10:24 PM

Maybe an employee can comment on that a bit more or in a future blog post about hardening KV. They could have internal failovers for R2 too by using its multiple regions and such

!protour (колобо́к)•6/17/25, 5:25 AM

Hey, quick question, the credentials for the KV when we move things to prod? Am I suppose to add it to the env vars, but how can I retrieve those

ddsgadget•6/19/25, 10:49 AM

I'm wondering why the dashboard for KV in cloudflare admin dashboard doesn't allow you to see the metadata for KV or edit this metadata? You can see the key and value, and edit those, but not the metadata. Is there anything in the works to fix this dashboard and expose the metadata because this makes the dashboard kind of useless for many KV applications, Thanks.

Zzecanard I'm having an issue with the KV cache. Up until recently (sometime after mid-Feb...

Abdallah Abis•6/19/25, 8:56 PM

I'm running into the same issue. Did you figure out a fix @@zecanard ?

Abdallah Abis•6/19/25, 9:07 PM

Hey folks,
We accidentally set the cacheTtlcacheTtl for a KV.get()KV.get() to 300000s300000s (~3.5 days) instead of 300s300s (5min). That resulted in caching the values and seeing stale data for longer than expected.
We went back and updated the cacheTtlcacheTtl to be 300300 after a couple days when we realized there's an issue with stale data, but we are still seeing stale data regardless. The fix was merged more than 2h ago, and fetched some entries to reset the cacheTtl as the docs say here, but no luck.
Any idea how to fix this?

The effective cacheTtl of an already cached item can be reduced by getting it again with a lower cacheTtl. For example, if you did NAMESPACE.get(key, {cacheTtl: 86400}) but later realized that caching for 24 hours was too long, you could NAMESPACE.get(key, {cacheTtl: 300}) or even NAMESPACE.get(key) and it would check for newer data to respect the provided cacheTtl, which defaults to 60 seconds.

Cloudflare Docs

Read key-value pairs

To get the value for a given key, call the get() method of the KV binding on any KV namespace you have bound to your Worker code:

AAbdallah Abis Hey folks, We accidentally set the `cacheTtl` for a `KV.get()` to `300000s` (~3....

zecanard•6/20/25, 7:20 AM

I think it was Micky who created a ticket to look into this for me. The problem seemed to resolve itself after two weeks, then everything was working fine for a while. However, recently I made another update to my KV, and the problem reoccurred for a few days. So it doesn't seem to be fixed at all.

Zzecanard I think it was Micky who created a ticket to look into this for me. The problem ...

Abdallah Abis•6/20/25, 1:24 PM

Wow! 2 weeks!! OMG

.
Did you set the cacheTtl to 2 weeks that's why the cache lasted that long, or 2 weeks is just a random thing?

AAbdallah Abis Wow! 2 weeks!! OMG 🤯 . Did you set the cacheTtl to 2 weeks that's why the cache...

zecanard•6/20/25, 6:47 PM

It was just random. I use a TTL of one year, and expire my KV cache whenever I update the values.

omortis•6/22/25, 4:07 PM

So I get an error trying to use --remote--remote:

✘ [ERROR] In development, you should use a separate kv namespace than the one you'd use in production. Please create a new kv namespace with "wrangler kv namespace create <name> --preview" and add its id as preview_id to the kv_namespace "CROM" in your wrangler.json file

And I follow the instructions and update my wrangler.jsonc:

  "kv_namespaces": [
    {
      "binding": "CROM",
      "id": "cda6c32602a643b1ac92d1f8bac5abff",
      "preview_id": "721f44f1924843a68312f39dc99192a1"
    }
  ]

  "kv_namespaces": [
    {
      "binding": "CROM",
      "id": "cda6c32602a643b1ac92d1f8bac5abff",
      "preview_id": "721f44f1924843a68312f39dc99192a1"
    }
  ]

And I still get the same error? Am I missing something?

omortis•6/22/25, 4:07 PM

I am trying to run the code with npx wrangler dev --remotenpx wrangler dev --remote

omortis•6/22/25, 4:09 PM

CROM and CROM_preview both exist in my dashboard as expected.

omortis•6/22/25, 4:11 PM

Got it. Need to build first. I guess --remote requires the built project.

MY FRIEND (Thomas) ♥♥•6/26/25, 11:06 AM

hello

koraytaylan•6/26/25, 7:34 PM

i was trying to build a POC where i wanted to add items as a tree. keys would be /master/child1, /master/child2 and then i would call list with /master/ prefix to get the child item keys and since the doc says list will always return keys in alphabetical order, i was hoping when delete child1, the next list1 call would return /master/child2 as the first item. however, for about a minute the list calls keep returning /master/child1. is this the expected behavior? i was making the list calls from within a workflow if that might be a factor somehow. replaced kv calls with a simple insert, select for d1 in the very same setup and the problem went away which made me think there must be some sort of caching for kv then.

Roz•7/2/25, 1:20 PM

hey guys
I was wondering if there's a way, possibly through a support ticket, to get a dump of a KV namespace
I have tried exporting the data via a script, but since the API only allows reading 100 records at a time, and I have 1.96M keys, it's gonna take ages

Is there a better way?

Dubz•7/5/25, 10:27 PM

Does KV have any native backup systems by chance or is it expected to do this on your own?

vmg•7/10/25, 3:49 PM

im working on a monorepo with multiple workers, how do i get kv to work across workers with the same namespace in development?

ie, if I post something to kv in worker 1, can i get it in worker 2?

Vvmg im working on a monorepo with multiple workers, how do i get kv to work across w...

Hello, I’m Allie!•7/10/25, 4:50 PM

Use the same directory via —persist-to

Dubz•7/10/25, 7:07 PM

Is there any native way to rename ~~containers~~ namespaces? I know it's referenced by ID and names whatever in workers, but for the sake of naming consistency it would be nice if I could change the name on the dashboard but not the ID.

DDubz Is there any native way to rename ~~containers~~ namespaces? I know it's referen...

Hello, I’m Allie!•7/11/25, 12:45 AM

Did you mean to ask in #containers-beta ?

I think the issue is that a lot of backend config stuff is stored in KV in US Central, which also ha

Similar Threads

Similar Threads

Similar Threads