With storage being replicated across all edges, or am I misunderstanding the premise behind R2?
With storage being replicated across all edges, or am I misunderstanding the premise behind R2?
II believe some replication was the plan. Not every colo otherwise that would get a bit out of hand (imagine storing a TB in every colo, that would be 350+ TB).
IVitali could confirm though.
MRight definitely don’t expect it for the smaller colo’s! Just in general at least some in like West US, Central, South, East US, + EU regions + KR/JP
HOMG, why is my 5TB Minecraft Map File not being served from the Madagascar colo? Totally Unusable!
IMight have been related to #cloudflare-status 
BI think asking about whether we're doing global replication is the wrong mental model. I think there are 4 fundamental things you care about as a customer:
- Durability - if R2 says it stored something, what's the probability that I can read that data back at any arbitrary point in time in the future?
- Availability - am I able to read existing data / write new data even if Cloudflare internally is experiencing some kind of local issue (i.e. how well can we hide problems / bugs from you)
- Performance - What's the TTFB for reads, what's the read/write RPS, what's the write latency for small files
- Regulations - Do I have an easy way to make sure I'm compliant with local regulations.
BFor example, this is where public buckets comes in. If you are able to cache assets being served out of your R2 bucket, the latency starts to matter a whole heck of a lot less because cache is truly globally distributed (+ your costs also go down).
IThat makes a lot of sense, great way to think of it!
IIf you can, can you quickly review what the current status of each of those points are and which you're working on?
ISorry to make more work for you haha, just curious.
B- Durability: 11 9s
- Availability: We do have some failure modes we know about that we're not satisfied with & work is ongoing to hopefully make some meaningful improvements by end of year or early. I think industry standard is 3 9s of availability. Amazon is at 4 9s on their marketing page (not sure what their actual contract states). The internal metrics I've seen so far have us in a pretty healthy spot for the past 30 day average.
- Performance: TTFB times should have dropped for most uploads/downloads. According to the community https://status.svge.ms we've come down a bit (need to double-check my notes as to where we started during open beta). That status page needs to be contextualized in that it's looking at the worst-case global access time. The data is clearly on the east coast somewhere. We have additional low-level network performance optimization work happening in the next quarter & we hope to cut that worst case by another 25-50% (not much of a difference for most people since access I think tends to be localized). Public buckets will have faster & more stable TTFB times than the S3 endpoint even at launch. Long tail access will drop dramatically by the end of the year & we're hoping to feed this through to S3 as well. Long distance download speeds should be increasing in the next few weeks/month as the TCP stack is tuned on the underlying storage nodes. Concurrent uploads for multipart issue will hopefully be fixed this quarter
- Regulations: Have not started thinking about this.
24 B / 0 B / 5 MB | Files - R2/Worker
API | KV/Worker
Pages
Pages Functions
Worker
Redirects | KV/Worker
Embed Redirects | KV/Worker
API | KV/Worker
Pages
Pages Functions
Worker
Redirects | KV/Worker
Embed Redirects | KV/Worker
MDurability I feel like is a huge part of global replication for me to at least. If, for instance, the NY DC gets hit with some natural disaster in the east coast or god forbid an internal fire (see OVH I think it was last year), global [or at least some form of] replication would allow for rapid recovery. Of course, I suspect CF already takes routine backups of customer data in R2 or whatnot to help ensure the availability, but I think true replication on a multi-continent scale would definitely up the durability of the data overall.
Z99.999999999% (there are 11 nines there, if I counted correctly)
IWow this is an awesome overview, thanks! Very excited to see the improvemens the R2 team's working on
ZMight be good to pin that?
K
BWhat do you mean by "rapid recovery"?
BRelease just went out:
- Support the
r2_list_honor_includecompat flag coming up in an upcoming runtime release (default behavior as of 2022-07-14 compat date). Without that compat flag/date, list will continue to function implicitly asinclude: ['httpMetadata', 'customMetadata']regardless of what you specify. cf-create-bucket-if-missingcan be set on aPutObject/CreateMultipartUploadrequest to implicitly create the bucket if it doesn't exist.- Fix S3 compatability with MinIO client spec non-compliant XML for publishing multipart uploads. Any leading and trailing quotes in CompleteMultipartUpload are now optional and ignored as it seems to be the actual non-standard behavior AWS implements.
KI love how fast the small bugs or client incompatibilities get fixed - sure, being in beta makes it easier but R2 gets the releases out so quick
BWorkers makes CI/CD really simple
B(+ it's kind of Cloudflare's MO)
K
KDoes the R2 Worker use service environments or the usual (well, currently only way) Worker per environment?
KService environments looked cool but they've been disabled ever since I first thought of using them
BEnvironments in terms of
[env.X] in wrangler.toml? Or something else?KI'm not entirely sure if it ever landed in
So instead of having a Worker like
wrangler.toml- or maybe it did as unsafe.
So instead of having a Worker like
worker-dev.subdomain.workers.dev, it was dev.worker.subdomain.workers.devBWe use
[env.X]. We have dedicated zones for every single team member, a staging zone that tracks the latest shared branch, and a production zone that tracks the release branch (with CI managing releases to staging & production). E2E tests can run locally (against Miniflare), dev zones (automatically pushing your current code), staging (never publishes, just runs tests) and prod. Prod is locked down so only the manager & maybe whoever is on-call have accessBWe don't use
workers.dev. We have 2 zones - staging & prod. Staging hosts the staging worker & developer workers (with relevant routes & whatnot)KInteresting, thanks for the info! I should probably start properly writing tests for my Workers but never really have done. All I've actually done at work is get rid of Workers as opposed to add them so it's been awhile since I've actually needed a Worker in a production setting haha
BAlso CI regularly runs the e2e test suite now against staging & prod so we can track that on an ongoing basis.
BOur tests are written locally in NodeJS
BThey're just fetch calls against production. Most of the magic is in
beforeAll to instantiate Miniflare or set up the right credsBThere's also some additional magic integrating with Miniflare to support hitting subdomains on Miniflare to test things like the UI endpoint.
and there's an
and there's an
apigw entry in the mounts parameter we instantiate Miniflare with which lets us add some environment variable overrides that are preferred if set (which they're not in prod)
INo, there's no published stats. But there have been a few blog posts detailing how efficient Workers are (including how, because of their fast startup time, CF can run WAY more Workers than Lambda ever could containers).
IFor a couple reasons:
- Workers only get 128Mb of RAM which means they can pack them in (sometimes they can even get less RAM if an isolate is extremely popular)
- Because they start up within 5ms, CF can move your Worker and you won't notice a thing
- Because CF has 350 colos they can distribute the workload out way more than someone like AWS who has fewer, bigger colos
BPeople severely underestimate how many RPS Workers is handling globally and overestimate how much load their service adds incrementally. There's certainly more than a million workers instantiated globally simultaneously.
BFor example, Apple private relay runs on top of Workers.
BThe Cloudflare Blog
iCloud Private Relay is a new Internet privacy service from Apple that allows users to connect to the Internet and browse with Safari in a more secure and private way. Cloudflare is proud to work with Apple to operate portions of Private Relay infrastructure