Cloudflare Developers•2y ago

Service updates, rollouts, etc. - anything could cause temporary blips, that's not uncommon with any

Service updates, rollouts, etc. - anything could cause temporary blips, that's not uncommon with any online service.

JJames Service updates, rollouts, etc. - anything could cause temporary blips, that's n...

kev-ac•2/14/24, 10:36 PM

Yeah. Possibly just high load at the moment and the services did not scale fast enough

Unsmart•2/14/24, 10:37 PM

dash works perfectly fine for me atm just slow to load the accounts I have access to, so seems likely its a rollout of some sort that will self resolve :NODDER:

kev-ac•2/14/24, 10:38 PM

As we gather here right now: Tomorrow is CF's test of their disaster recovery process. There will be short API outages. Hopefully not more

https://www.cloudflarestatus.com/incidents/0j9638c9ddxm

Control Plane Test on Thursday, February 15th, 2024

kev-ac•2/14/24, 10:42 PM

The engineer who is "pulling the plug" on their core control plane facility must have some fun. I certainly would have

Walshy•2/14/24, 10:43 PM

There are two types of people in the world, those who love chaos and those who hate it

I think only the first type would have fun there haha

Walshy•2/14/24, 10:44 PM

but seriously, as mentioned in the message, the impact should be short and then smooth sailing. We've worked months to make sure we're in a good state and now we're putting it to the test

kev-ac•2/14/24, 10:44 PM

Best of both worlds: Chaos that someone created intentionally and with permission

Walshy•2/14/24, 10:44 PM

the best type of chaos

Original message was deleted

kev-ac•2/14/24, 10:46 PM

Definitely, having to build and maintain systems and software on the scale of Cloudflare is scary

Original message was deleted

kev-ac•2/14/24, 10:49 PM

The regex bug?

Original message was deleted

kev-ac•2/14/24, 10:50 PM

CFs energy bill in that time:

Having all CPUs pinned at 100%

kev-ac•2/14/24, 10:53 PM

And don't forget reading the post mortems after bigger incidents. Getting some background about what happened internally that lead to some service fail catastrophically is fascinating.

Original message was deleted

kev-ac•2/14/24, 10:55 PM

Was that after the regex / WAF incident?

kev-ac•2/14/24, 10:55 PM

Oh I see, just googled it. Time passes quick...

Walshy•2/14/24, 10:56 PM

i got paged for that :meowupsidedown:

Walshy•2/14/24, 10:57 PM

my first week oncall iirc

WWalshy my first week oncall iirc

kev-ac•2/14/24, 10:57 PM

Good way to start haha

EErisa Try this - https://developers.cloudflare.com/pages/how-to/deploy-a-wordpress-sit...

Graeme•2/14/24, 10:57 PM

Thank you

Chaika•2/14/24, 10:58 PM

that was when I first learned about MCPs and maybe the first blog to mention them?

Walshy•2/14/24, 10:58 PM

it was the first blog yep

Chaika•2/14/24, 10:59 PM

funny thing that PoPs with more then one colo are multiple colo pops but may not be mcps ;p

kev-ac•2/14/24, 11:02 PM

@Walshy | Deploying Maybe it was said in a blog post or it may be a internal secret: When some service goes rogue there were few instances where CF issued a global terminate on that service according to the blog posts. Is that something multiple people need to "hit go on" or are these just team members in higher levels? Having the power to globally kill services is something.

kev-ac•2/14/24, 11:03 PM

Thinking theoretically about how I could implement processes for systems that allow that kind of action and who can do it.

Walshy•2/14/24, 11:03 PM

Not multiple people but a member of the team (and related roles) can globally "killswitch" their service in an incident

WWalshy Not multiple people but a member of the team (and related roles) can globally "k...

kev-ac•2/14/24, 11:04 PM

Where we'll be back to the root of hate or love chaos...

Walshy•2/14/24, 11:06 PM

#ILoveChaos

Chaika•2/14/24, 11:06 PM

it's not a true test unless you go and unplug the entire DC again ;p

Walshy•2/14/24, 11:07 PM

i'll get DC team on it

CChaika it's not a true test unless you go and unplug the entire DC again ;p

kev-ac•2/14/24, 11:08 PM

Why do I have the scene from South Park in my head where Kyle disconnects "the internet" while reading this message? Must kindof feel like it. For those who remember: https://www.youtube.com/watch?v=ckIMuvumYrg

Original message was deleted

Chaika•2/14/24, 11:10 PM

iirc it just looks up region1.v2.argotunnel.com / region2.v2.argotunnel.com and tries a bunch of the IPs

CChaika it's not a true test unless you go and unplug the entire DC again ;p