* Durability: 11 9s * Availability: We do have some failure modes we know about that we're n

Durability: 11 9s
Availability: We do have some failure modes we know about that we're not satisfied with & work is ongoing to hopefully make some meaningful improvements by end of year or early. I think industry standard is 3 9s of availability. Amazon is at 4 9s on their marketing page (not sure what their actual contract states). The internal metrics I've seen so far have us in a pretty healthy spot for the past 30 day average.
Performance: TTFB times should have dropped for most uploads/downloads. According to the community https://status.svge.ms we've come down a bit (need to double-check my notes as to where we started during open beta). That status page needs to be contextualized in that it's looking at the worst-case global access time. The data is clearly on the east coast somewhere. We have additional low-level network performance optimization work happening in the next quarter & we hope to cut that worst case by another 25-50% (not much of a difference for most people since access I think tends to be localized). Public buckets will have faster & more stable TTFB times than the S3 endpoint even at launch. Long tail access will drop dramatically by the end of the year & we're hoping to feed this through to S3 as well. Long distance download speeds should be increasing in the next few weeks/month as the TCP stack is tuned on the underlying storage nodes. Concurrent uploads for multipart issue will hopefully be fixed this quarter
Regulations: Have not started thinking about this.

As you can tell, performance is a primary focus for this quarter. Availability is a parallel effort with the bulk of the gains probably H1 2023 but lots of interim gains along the way. Unfortunately availability is a very laggy and sporadic indicator to track (5 9s one month, 2 9s the next). The only thing you can really track is day to day availability improvements but not your robustness to outages.

svge.ms

24 B / 0 B / 5 MB | Files - R2/Worker
API | KV/Worker
Pages
Pages Functions
Worker
Redirects | KV/Worker
Embed Redirects | KV/Worker

Bbeelzabub * **Durability**: 11 9s * **Availability**: We do have some failure modes we kno...

Hello, I’m Allie!•7/6/22, 7:43 PM

11 9s?

musicman123 — FJLaboratories•7/6/22, 7:46 PM

Durability I feel like is a huge part of global replication for me to at least. If, for instance, the NY DC gets hit with some natural disaster in the east coast or god forbid an internal fire (see OVH I think it was last year), global [or at least some form of] replication would allow for rapid recovery. Of course, I suspect CF already takes routine backups of customer data in R2 or whatnot to help ensure the availability, but I think true replication on a multi-continent scale would definitely up the durability of the data overall.

HHello, I’m Allie!11 9s?

zegevlier•7/6/22, 7:49 PM

99.999999999% (there are 11 nines there, if I counted correctly)

Bbeelzabub * **Durability**: 11 9s * **Availability**: We do have some failure modes we kno...

Isaac McFadyen•7/6/22, 7:53 PM

Wow this is an awesome overview, thanks! Very excited to see the improvemens the R2 team's working on

zegevlier•7/6/22, 7:55 PM

Might be good to pin that?

kian•7/6/22, 10:04 PM

https://github.com/cloudflare/cloudflare-docs/pull/5058/files

GitHub

[R2] Document cf-create-bucket-if-missing by vlovich · Pull Request...

kian•7/6/22, 10:04 PM

Mmusicman123 — FJLaboratories Durability I feel like is a huge part of global replication for me to at least. ...

beelzabubOP•7/6/22, 10:04 PM

What do you mean by "rapid recovery"?

Kkian https://github.com/cloudflare/cloudflare-docs/pull/5058/files

beelzabubOP•7/6/22, 10:05 PM

Release just went out:

Support the compat flag coming up in an upcoming runtime release (default behavior as of 2022-07-14 compat date). Without that compat flag/date, list will continue to function implicitly as regardless of what you specify.
can be set on a / request to implicitly create the bucket if it doesn't exist.
Fix S3 compatability with MinIO client spec non-compliant XML for publishing multipart uploads. Any leading and trailing quotes in CompleteMultipartUpload are now optional and ignored as it seems to be the actual non-standard behavior AWS implements.

kian•7/6/22, 10:12 PM

I love how fast the small bugs or client incompatibilities get fixed - sure, being in beta makes it easier but R2 gets the releases out so quick

beelzabubOP•7/6/22, 10:13 PM

Workers makes CI/CD really simple

beelzabubOP•7/6/22, 10:13 PM

(+ it's kind of Cloudflare's MO)

kian•7/6/22, 10:15 PM

kian•7/6/22, 10:16 PM

Does the R2 Worker use service environments or the usual (well, currently only way) Worker per environment?

kian•7/6/22, 10:16 PM

Service environments looked cool but they've been disabled ever since I first thought of using them

beelzabubOP•7/6/22, 10:18 PM

Environments in terms of in wrangler.toml? Or something else?

kian•7/6/22, 10:19 PM

I'm not entirely sure if it ever landed in

wrangler.toml

wrangler.toml

or maybe it did as unsafe.

https://developers.cloudflare.com/workers/learning/using-services/>

So instead of having a Worker like , it was

beelzabubOP•7/6/22, 10:20 PM

We use . We have dedicated zones for every single team member, a staging zone that tracks the latest shared branch, and a production zone that tracks the release branch (with CI managing releases to staging & production). E2E tests can run locally (against Miniflare), dev zones (automatically pushing your current code), staging (never publishes, just runs tests) and prod. Prod is locked down so only the manager & maybe whoever is on-call have access

beelzabubOP•7/6/22, 10:21 PM

We don't use

workers.dev

workers.dev

. We have 2 zones - staging & prod. Staging hosts the staging worker & developer workers (with relevant routes & whatnot)

kian•7/6/22, 10:23 PM

Interesting, thanks for the info! I should probably start properly writing tests for my Workers but never really have done. All I've actually done at work is get rid of Workers as opposed to add them so it's been awhile since I've actually needed a Worker in a production setting haha

beelzabubOP•7/6/22, 10:24 PM

Also CI regularly runs the e2e test suite now against staging & prod so we can track that on an ongoing basis.

beelzabubOP•7/6/22, 10:24 PM

Our tests are written locally in NodeJS

beelzabubOP•7/6/22, 10:24 PM

They're just fetch calls against production. Most of the magic is in to instantiate or set up the right creds

beelzabubOP•7/6/22, 10:27 PM

There's also some additional magic integrating with Miniflare to support hitting subdomains on Miniflare to test things like the UI endpoint.

and there's an entry in the parameter we instantiate Miniflare with which lets us add some environment variable overrides that are preferred if set (which they're not in prod)

GGHOST is there any blogs about the amount of servers cf has in any given DC? because i...

Isaac McFadyen•7/6/22, 10:44 PM

No, there's no published stats. But there have been a few blog posts detailing how efficient Workers are (including how, because of their fast startup time, CF can run WAY more Workers than Lambda ever could containers).

Isaac McFadyen•7/6/22, 10:45 PM

For a couple reasons:

Workers only get 128Mb of RAM which means they can pack them in (sometimes they can even get less RAM if an isolate is extremely popular)
Because they start up within 5ms, CF can move your Worker and you won't notice a thing
Because CF has 350 colos they can distribute the workload out way more than someone like AWS who has fewer, bigger colos

GGHOST is there any blogs about the amount of servers cf has in any given DC? because i...

beelzabubOP•7/6/22, 10:51 PM

People severely underestimate how many RPS Workers is handling globally and overestimate how much load their service adds incrementally. There's certainly more than a million workers instantiated globally simultaneously.

beelzabubOP•7/6/22, 10:53 PM

For example, Apple private relay runs on top of Workers.

beelzabubOP•7/6/22, 10:53 PM

https://blog.cloudflare.com/icloud-private-relay/

The Cloudflare Blog

iCloud Private Relay: information for Cloudflare customers

iCloud Private Relay is a new Internet privacy service from Apple that allows users to connect to the Internet and browse with Safari in a more secure and private way. Cloudflare is proud to work with Apple to operate portions of Private Relay infrastructure

Unsmart•7/6/22, 10:53 PM

Does workers have the same limitation as aws lambda in that any one process can only be handling one request, and if there are multiple at the same time it runs a new script?

beelzabubOP•7/6/22, 10:55 PM

No. We can handle very many connections simultaneously within 1 script. We do want to start isolating the scripts so that they can't access each other's memory (cross request sharing of globals are a common source of pain & can cause vulnerabilities in your code if you're not careful) but there's R&D surgery deep in the bowels of V8 that needs to be done to enable that.

beelzabubOP•7/6/22, 10:55 PM

If a script is handling too many requests within one isolate, then additional isolates get spun up to handle the load (up to a limit)

beelzabubOP•7/6/22, 10:56 PM

There's roughly only 1 Cloudflare Workers process for all scripts that are running (there's technically more for security reasons but effectively just 1 running the scripts)

Unsmart•7/6/22, 10:58 PM

Interesting, very cool stuff being pumped out here had to start looking into everything, DO & alarms, R2, etc

Bbeelzabub People severely underestimate how many RPS Workers is handling globally and over...

Isaac McFadyen•7/6/22, 11:10 PM

Assuming 128mb/Worker, then that's 128 TERABYTES of RAM alone. Wow.

Isaac McFadyen•7/6/22, 11:11 PM

Think it's 128mb/isolate so probably less, but also there's probably more than a million I'm guessing.

Bbeelzabub What do you mean by "rapid recovery"?

musicman123 — FJLaboratories•7/6/22, 11:17 PM

More just thinking of the "time till everything's back to operational" or "time till data is recovered." Moving from primarily an on-prem mindset where we had our data in a few fixed locations, if those locations got hit by something, we focused on being able to reroute things back to another location that had the data intact.

I know cloud storage is a different model entirely though, but my mind is still stuck in the older days.

HHello, I’m Allie!That’s only if you don’t have a business license. If you do, there are almost 40...

yinxingmaiming6409•7/7/22, 1:58 AM

Business authorization costs at least $5000, which is difficult for ordinary people to afford

Yyinxingmaiming6409 Business authorization costs at least $5000, which is difficult for ordinary peo...

Isaac McFadyen•7/7/22, 2:10 AM

Yes, that's true. The point of the observation was that it's not Cloudflare's fault that routing in China goes to the US without a license, that's just the way licensing requirements on the China Network work.

IIsaac McFadyen Yes, that's true. The point of the observation was that it's not Cloudflare's fa...

yinxingmaiming6409•7/7/22, 4:41 AM

I know, but the price is too expensive.

XdGoldenTiger•7/7/22, 4:51 AM

That is a reasonable price. I dont think people understand that a Business plan is for basically (Not saying that it cant be used for personal use) for businesses that are big, who earn alot, to be able to maintain operating systems.
If you dont like the price. Just use a cheaper/free option? That simple lol.

Jacob•7/7/22, 2:43 PM

Migrated my Minecraft Overviewer maps to R2+B2 (instead of Spaces+B2) and wow, it's so fast!! Loving the low latency https://mc-maps.geostyx.com/maps/servers/ftb-ultimate-ae-2022/index.html#/-4864/64/2560/-7/FTBUltimateAE2022%20-%20overworld/survivalday
Here's the handler for it that races R2 and B2 so I can arbitrarily store data in both storage providers: https://gist.github.com/jahands/a9625d2ac6553cce6a86aeb6d9cb8a18

Jacob•7/7/22, 2:50 PM

It's loading about as fast as Google Cloud Storage, which is fantastic

JJacob Migrated my Minecraft Overviewer maps to R2+B2 (instead of Spaces+B2) and wow, i...

James•7/7/22, 3:41 PM

This is sweet

Mmusicman123 — FJLaboratories More just thinking of the "time till everything's back to operational" or "time ...

beelzabubOP•7/7/22, 4:58 PM

Externally that’s visible as availability not durability and is something for us to manage. Replicating in geographically distant regions is just one strategy.

JJacob Migrated my Minecraft Overviewer maps to R2+B2 (instead of Spaces+B2) and wow, i...

MyZeD•7/7/22, 5:19 PM

I use a external small Webhosting + Vector Tiles Maps (Not Minecraft) and cache it through Proxyed CF.
So i dont need Workers

Jacob•7/7/22, 5:48 PM

Nice! I've found it's a challenge to efficiently host Leaflet maps at scale. I have about 37 million images taking up about 2TB of space. I used to host on a VPS, but quickly had issues with speed in different regions, and SSDs get expensive

* **Durability**: 11 9s * **Availability**: We do have some failure modes we know about that we're n

* Durability: 11 9s * Availability: We do have some failure modes we know about that we're n