Cloudflare Developers•7mo ago

i don't use supabase

ddsgadgetOP•6/12/25, 6:39 PM

best to use managed database services directly. I use digital ocean

Dddsgadget i don't use supabase

AZipr•6/12/25, 6:40 PM

Nice. But DO also has some bad reputation. Not meeting SLAs and all

juanferreras•6/12/25, 6:41 PM

some say AWS is having issues too. We'd know if... any of the big providers would bother updating their status page too

ddsgadgetOP•6/12/25, 6:41 PM

DO has been fine. It's unlikely DO and Cloudflare go down at the same time. Anyway, I've been moving many services from DO to Cloudflare and now I'll have to just keep both b/c having one point of failure in Cloudflare was just stupid decision

Dddsgadget this just solidifies my decision to just go back to storage with postgres and re...

Sam•6/12/25, 7:53 PM

We moved all our app code off workers because of D1/DurObj/KV reliability concerns, but are still using KV for versioning/routing. This may be the push to finally switch to versioned workers (wrote it before those were a thing).

Is KV us-central-1?

ddsgadgetOP•6/12/25, 7:57 PM

They say services recovering, but I dn't see it all. this is a total disaster with KV and completely removes my faith in it as reliable storage. KV is probably best used as a backup for another service.

Sam•6/12/25, 7:58 PM

ZeroTrust/Access definitely recovered (I can reach internal services again), but yeah definitely not KV

parisholley•6/12/25, 8:06 PM

i was under the impression CF ran its own pops/colos, the fact KV is 100% reliant on GCP is concerning

juanferreras•6/12/25, 8:07 PM

KV started responding successfully to me

threecreepio•6/12/25, 8:07 PM

ay seems to be working now

Pparisholley i was under the impression CF ran its own pops/colos, the fact KV is 100% relian...

1984 Ford Laser•6/12/25, 8:07 PM

I think the issue is that a lot of backend config stuff is stored in KV in US Central, which also happened to be the location hardest hit by the looks of it

parisholley•6/12/25, 8:08 PM

didn't last big outage show holes in their config architecture? thought that was all being mitigated

Sam•6/12/25, 8:09 PM

Error listing KV entries: KV GET failed: 503 Service temporarily unavailableError listing KV entries: KV GET failed: 503 Service temporarily unavailable Hopefully it hits me soon

Sam•6/12/25, 8:10 PM

ayyy there it is

11984 Ford Laser I think the issue is that a lot of backend config stuff is stored in KV in US Ce...

Chaika•6/12/25, 8:12 PM

important backend stuff/anything config should be QS (Quicksilver, Cf's own replicated edge kv db)

Chaika•6/12/25, 8:12 PM

that's why websites/cdn/proxy continued to work fine

Chaika•6/12/25, 8:13 PM

KV is (clearly now) GCP dependent, and CF Access, dashboard, etc, depend onKV

ddsgadgetOP•6/12/25, 8:16 PM

all my kv are still down, i guess they only recover KV by geographic location or something?

Dddsgadget all my kv are still down, i guess they only recover KV by geographic location or...

Chaika•6/12/25, 8:18 PM

here's def still some errors, thundering herd problem and all that, recovery is slow

Chaika•6/12/25, 9:04 PM

All Cloudflare services have been restored and are now fully operational. We are moving the incident to Monitoring while we watch platform metrics to confirm sustained stability.

Reikoshea•6/12/25, 9:12 PM

all my sites use KV, and they're all back but 1. For some reason that 1 is still throwing the KV 503.

Reikoshea•6/12/25, 9:13 PM

throws it on both the site and the in the worker ide.

ddsgadgetOP•6/12/25, 9:18 PM

for me it's starting to come back. One thing to say about Cloudflare is that they "eat their own cooking", so if something like KV goes down, even they are screwed, and they've got a lot to lose. So in that sense, issues with something like KV are attended to almost immediately. That being said, this was a pretty bad outage and for me at least, I am reluctant to rely on it as much as I have

Reikoshea•6/12/25, 9:19 PM

The same is true for pretty much everyone. When kenisis went down in AWS a couple years ago, it took out most of AWS's provisioning and updating tools

ddsgadgetOP•6/13/25, 1:32 AM

Just started getting KV failures again. This thing is not stable.

1984 Ford Laser•6/13/25, 2:16 AM

Seems to be working fine for me

AZipr•6/13/25, 4:07 AM

It was a major outage. CF GCP and AWS were affected.

https://www.livemint.com/news/us-news/netizens-claim-major-internet-outage-caused-by-cyberattack-all-details-on-cloudflare-gcp-downtime-here-11749765997009.html#

Chaika•6/13/25, 4:14 AM

nah, aws was fine. It was just GCP failing which cascaded into CF/Spotify/etc (and CF's customers, etc)

AAZipr It was a major outage. CF GCP and AWS were affected. https://www.livemint.com...

crossbeau•6/13/25, 4:34 AM

Author is drunk. They wrote wednesday many times

AZipr•6/13/25, 5:07 AM

:bibicat:

thomasgauvin•6/13/25, 2:40 PM

Hi folks, we published an update on the service outage yesterday here: https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/

MMallissin I was reading the post mortem and I'm really afraid to ask this, but was Cloudfl...

Chaika•6/13/25, 6:28 PM

lol def not Google Sheets. They were using one of Google Cloud's products. They weren't specific on which ones they are dependent on, but you could make an educated guess

Chaika•6/13/25, 6:30 PM

Why are all the explanations being so vague?

Transparency is nice and it'd be cool to see a blog post about what exactly is used under the hood, but I mean for an incident report the bottom line is just: Cloudflare's products failed for hours due to an outage. Themselves, upstream or not, it's their responsibility to make systems reliable and not have downtime, and they detailed they're working towards that

imo you can't (and they didn't, they didn't even name GCP) just blame partners for outages, when you're the one responsible for picking them and ultimately providing the service.

MMallissin I was teasing and yes, can guess which particular service was used, but I still ...

Chaika•6/13/25, 10:22 PM

I would note that they said at one point they had another fail-over/partner

Workers KV removed a storage provider as we worked to re-architect KV’s backend

Moving the object store inhouse does not solve the issue of lack of fall-over. What if a problem with R2 causes a similar issue in the future?

I don't think they were really clear fully on their plans, maybe they plan on their own + GCP as a backup. R2 is their service/internal to them, even if they planned on going with just R2 if there's any issues it's internal to them/for them to fix. Basically them being in control of more the infra in the same way GCS does for example

Chaika•6/13/25, 10:24 PM

Maybe an employee can comment on that a bit more or in a future blog post about hardening KV. They could have internal failovers for R2 too by using its multiple regions and such

!protour (колобо́к)•6/17/25, 5:25 AM

Hey, quick question, the credentials for the KV when we move things to prod? Am I suppose to add it to the env vars, but how can I retrieve those

ddsgadgetOP•6/19/25, 10:49 AM

I'm wondering why the dashboard for KV in cloudflare admin dashboard doesn't allow you to see the metadata for KV or edit this metadata? You can see the key and value, and edit those, but not the metadata. Is there anything in the works to fix this dashboard and expose the metadata because this makes the dashboard kind of useless for many KV applications, Thanks.

Zzecanard I'm having an issue with the KV cache. Up until recently (sometime after mid-Feb...

Abdallah Abis•6/19/25, 8:56 PM

I'm running into the same issue. Did you figure out a fix @@zecanard ?

Abdallah Abis•6/19/25, 9:07 PM

Hey folks,
We accidentally set the cacheTtlcacheTtl for a KV.get()KV.get() to 300000s300000s (~3.5 days) instead of 300s300s (5min). That resulted in caching the values and seeing stale data for longer than expected.
We went back and updated the cacheTtlcacheTtl to be 300300 after a couple days when we realized there's an issue with stale data, but we are still seeing stale data regardless. The fix was merged more than 2h ago, and fetched some entries to reset the cacheTtl as the docs say here, but no luck.
Any idea how to fix this?

The effective cacheTtl of an already cached item can be reduced by getting it again with a lower cacheTtl. For example, if you did NAMESPACE.get(key, {cacheTtl: 86400}) but later realized that caching for 24 hours was too long, you could NAMESPACE.get(key, {cacheTtl: 300}) or even NAMESPACE.get(key) and it would check for newer data to respect the provided cacheTtl, which defaults to 60 seconds.

Cloudflare Docs

Read key-value pairs

To get the value for a given key, call the get() method of the KV binding on any KV namespace you have bound to your Worker code:

AAbdallah Abis Hey folks, We accidentally set the `cacheTtl` for a `KV.get()` to `300000s` (~3....

zecanard•6/20/25, 7:20 AM

I think it was Micky who created a ticket to look into this for me. The problem seemed to resolve itself after two weeks, then everything was working fine for a while. However, recently I made another update to my KV, and the problem reoccurred for a few days. So it doesn't seem to be fixed at all.

Zzecanard I think it was Micky who created a ticket to look into this for me. The problem ...

Abdallah Abis•6/20/25, 1:24 PM

Wow! 2 weeks!! OMG

.
Did you set the cacheTtl to 2 weeks that's why the cache lasted that long, or 2 weeks is just a random thing?

i don't use supabase

Similar Threads

Similar Threads

Similar Threads