That’s only if you don’t have a business license. If you do, there are almost 40 colos in China that
That’s only if you don’t have a business license. If you do, there are almost 40 colos in China that your request can be served from
MWhat is the state of R2’s global edge distribution these days anyways? Of course it’s in no way full coverage yet, but has R2 been replicated to the EU and greater Asia regions yet?
MLast I recall, most of the storage has been in the USA but wondering if that’s changed recently
KIf you upload/copy an object from the EU then it'll be stored in EU
MOoo got it! Is this going to be the final structure then, with different regions having their data localized only to that region?
MBecause I thought the original push was going for true global data distribution
MWith storage being replicated across all edges, or am I misunderstanding the premise behind R2?
II believe some replication was the plan. Not every colo otherwise that would get a bit out of hand (imagine storing a TB in every colo, that would be 350+ TB).
IVitali could confirm though.
MRight definitely don’t expect it for the smaller colo’s! Just in general at least some in like West US, Central, South, East US, + EU regions + KR/JP
HOMG, why is my 5TB Minecraft Map File not being served from the Madagascar colo? Totally Unusable!
IMight have been related to #cloudflare-status 
BI think asking about whether we're doing global replication is the wrong mental model. I think there are 4 fundamental things you care about as a customer:
- Durability - if R2 says it stored something, what's the probability that I can read that data back at any arbitrary point in time in the future?
- Availability - am I able to read existing data / write new data even if Cloudflare internally is experiencing some kind of local issue (i.e. how well can we hide problems / bugs from you)
- Performance - What's the TTFB for reads, what's the read/write RPS, what's the write latency for small files
- Regulations - Do I have an easy way to make sure I'm compliant with local regulations.
BFor example, this is where public buckets comes in. If you are able to cache assets being served out of your R2 bucket, the latency starts to matter a whole heck of a lot less because cache is truly globally distributed (+ your costs also go down).
IThat makes a lot of sense, great way to think of it!
IIf you can, can you quickly review what the current status of each of those points are and which you're working on?
ISorry to make more work for you haha, just curious.
B- Durability: 11 9s
- Availability: We do have some failure modes we know about that we're not satisfied with & work is ongoing to hopefully make some meaningful improvements by end of year or early. I think industry standard is 3 9s of availability. Amazon is at 4 9s on their marketing page (not sure what their actual contract states). The internal metrics I've seen so far have us in a pretty healthy spot for the past 30 day average.
- Performance: TTFB times should have dropped for most uploads/downloads. According to the community https://status.svge.ms we've come down a bit (need to double-check my notes as to where we started during open beta). That status page needs to be contextualized in that it's looking at the worst-case global access time. The data is clearly on the east coast somewhere. We have additional low-level network performance optimization work happening in the next quarter & we hope to cut that worst case by another 25-50% (not much of a difference for most people since access I think tends to be localized). Public buckets will have faster & more stable TTFB times than the S3 endpoint even at launch. Long tail access will drop dramatically by the end of the year & we're hoping to feed this through to S3 as well. Long distance download speeds should be increasing in the next few weeks/month as the TCP stack is tuned on the underlying storage nodes. Concurrent uploads for multipart issue will hopefully be fixed this quarter
- Regulations: Have not started thinking about this.
24 B / 0 B / 5 MB | Files - R2/Worker
API | KV/Worker
Pages
Pages Functions
Worker
Redirects | KV/Worker
Embed Redirects | KV/Worker
API | KV/Worker
Pages
Pages Functions
Worker
Redirects | KV/Worker
Embed Redirects | KV/Worker
MDurability I feel like is a huge part of global replication for me to at least. If, for instance, the NY DC gets hit with some natural disaster in the east coast or god forbid an internal fire (see OVH I think it was last year), global [or at least some form of] replication would allow for rapid recovery. Of course, I suspect CF already takes routine backups of customer data in R2 or whatnot to help ensure the availability, but I think true replication on a multi-continent scale would definitely up the durability of the data overall.
Z99.999999999% (there are 11 nines there, if I counted correctly)
IWow this is an awesome overview, thanks! Very excited to see the improvemens the R2 team's working on
ZMight be good to pin that?
K
BWhat do you mean by "rapid recovery"?
BRelease just went out:
- Support the
r2_list_honor_includecompat flag coming up in an upcoming runtime release (default behavior as of 2022-07-14 compat date). Without that compat flag/date, list will continue to function implicitly asinclude: ['httpMetadata', 'customMetadata']regardless of what you specify. cf-create-bucket-if-missingcan be set on aPutObject/
request to implicitly create the bucket if it doesn't exist.CreateMultipartUpload- Fix S3 compatability with MinIO client spec non-compliant XML for publishing multipart uploads. Any leading and trailing quotes in CompleteMultipartUpload are now optional and ignored as it seems to be the actual non-standard behavior AWS implements.
KI love how fast the small bugs or client incompatibilities get fixed - sure, being in beta makes it easier but R2 gets the releases out so quick
BWorkers makes CI/CD really simple
B(+ it's kind of Cloudflare's MO)
K
KDoes the R2 Worker use service environments or the usual (well, currently only way) Worker per environment?
KService environments looked cool but they've been disabled ever since I first thought of using them
BEnvironments in terms of
[env.X] in wrangler.toml? Or something else?KI'm not entirely sure if it ever landed in
So instead of having a Worker like
wrangler.toml- or maybe it did as unsafe.
So instead of having a Worker like
worker-dev.subdomain.workers.dev, it was dev.worker.subdomain.workers.devBWe use
[env.X]. We have dedicated zones for every single team member, a staging zone that tracks the latest shared branch, and a production zone that tracks the release branch (with CI managing releases to staging & production). E2E tests can run locally (against Miniflare), dev zones (automatically pushing your current code), staging (never publishes, just runs tests) and prod. Prod is locked down so only the manager & maybe whoever is on-call have accessBWe don't use
workers.devInteresting, thanks for the info! I should probably start properly writing tests for my Workers but never really have done. All I've actually done at work is get rid of Workers as opposed to add them so it's been awhile since I've actually needed a Worker in a production setting haha
BAlso CI regularly runs the e2e test suite now against staging & prod so we can track that on an ongoing basis.
BOur tests are written locally in NodeJS
BThey're just fetch calls against production. Most of the magic is in
beforeAll to instantiate Miniflare or set up the right creds