Quick question, so what do you think about using R2 for an Unreal Engine S3 Shared Derived Data Cach

Quick question, so what do you think about using R2 for an Unreal Engine S3 Shared Derived Data Cache (https://docs.unrealengine.com/4.27/en-US/ProductionPipelines/DerivedDataCache/#q:whatisthes3ddc?) ? Will be be expensive due to a large ammount of requests?

Meowcat285OP•6/29/22, 9:18 PM

I am currently using a self hosted S3 compatible server for this, but the distrubited nature of R2 makes it very apealing

Meowcat285OP•6/29/22, 9:21 PM

Another worry is that it may not support the S3 API call ListObjects, but that seems to have a fallback

EErisa The way I understood the messages, it would be public through the CNAME (your cu...

beelzabub•6/29/22, 9:25 PM

You would actually be able to either or:

Totally public
Access restricted through an allow list of CNames (probably up to 100 or 1k - need to figure out what makes sense)

Totally public doesn't preclude putting a CName in front but then you have to contend with auth in the sense that if your public bucket ID leaks somehow, then you've got no auth. However, as long as you keep the public ID secret and use a Cloudflare-managed CName you should be "OK". It's security through obscurity but a Cloudflare CName will mask your public bucket ID. Public bucket IDs have a flat globally unique domain that you can't figure out the underlying account/bucket name for. The mapping back & forth is internal to Cloudflare. The public bucket is a randomly generated UUID string and indeterminable even if you somehow knew the account / bucket name (so all the community members posting their account IDs, it won't give attackers any information about what the public name of that bucket is & there's no way to brute force it).

MMeowcat285 Another worry is that it may not support the S3 API call ListObjects, but that s...

beelzabub•6/29/22, 9:26 PM

We definitely support ListObjects

MMeowcat285 Quick question, so what do you think about using R2 for an Unreal Engine S3 Shar...

beelzabub•6/29/22, 9:27 PM

How many requests are we talking about? Pricing is here: https://developers.cloudflare.com/r2/platform/pricing/

Pricing · Cloudflare R2 docs

Store large amounts of unstructured data without egress fees.

beelzabub•6/29/22, 9:27 PM

Putting cache in front of it would reduce the price since cached results wouldn't be billed.

Cyb3r-Jak3•6/29/22, 10:00 PM

Is there a plan to add R2 endpoint to the REST API?

beelzabub•6/29/22, 10:02 PM

By REST API, do you mean APIv4?

Cyb3r-Jak3•6/29/22, 10:03 PM

Yes

beelzabub•6/29/22, 10:04 PM

It's actually there already. That's what the UI uses underneath and it's fairly complete (the primary users are UI & wrangler). There aren't plans any time soon to actually stabilize and document it though.

Cyb3r-Jak3•6/29/22, 10:07 PM

Yeah I've been inspecting it but wanted to use terraform to manage buckets.

beelzabub•6/29/22, 10:11 PM

Which actions specifically do you need?

Cyb3r-Jak3•6/29/22, 10:37 PM

Pretty much just create and delete

beelzabub•6/29/22, 10:37 PM

I doubt we'd change that

Cyb3r-Jak3•6/29/22, 10:38 PM

And API token creation and delete

kian•6/29/22, 10:38 PM

I think the TF provider relies on them being in cloudflare-go

kian•6/29/22, 10:38 PM

Should(?) be easy enough to add those in though

Kkian I think the TF provider relies on them being in cloudflare-go

Cyb3r-Jak3•6/29/22, 10:38 PM

It does which requires them to have a public API

beelzabub•6/29/22, 10:38 PM

R2 API token creation/deletion is the same as stock API token creation/deletion. The only nuance is that the token id is sha256 of the normal token id

kian•6/29/22, 10:39 PM

My R2 has disappeared from the dashboard - huh

Kkian My R2 has disappeared from the dashboard - huh

Cyb3r-Jak3•6/29/22, 10:39 PM

https://giphy.com/gifs/FootballAustralia-soccer-referee-red-card-TbRwmI2fHg6ELPObHH

GiphyFootball Australia

kian•6/29/22, 10:39 PM

kian•6/29/22, 10:39 PM

and /flags/flags is 504ing

beelzabub•6/29/22, 10:39 PM

Uggh

kian•6/29/22, 10:39 PM

kian•6/29/22, 10:39 PM

Ooo, it's back now

beelzabub•6/29/22, 10:39 PM

We have a todo to disconnect the flipper flag

beelzabub•6/29/22, 10:40 PM

But we've been in release freeze for ~2 weeks

beelzabub•6/29/22, 10:40 PM

Next week is going to be a bunch of stuff landing all at once

kian•6/29/22, 10:40 PM

Seems like flipper might be a bit upset at the moment

kian•6/29/22, 10:40 PM

It's either super slow or, yeah, timeout

Bbeelzabub R2 API token creation/deletion is the same as stock API token creation/deletion....

Cyb3r-Jak3•6/29/22, 10:41 PM

This is probably be do-able but for R2 there needs to be a public API otherwise it gets stuck under https://github.com/cloudflare/cloudflare-go/issues?q=is%3Aissue+is%3Aopen+label%3Aworkflow%2Fpending-public-api

kian•6/29/22, 10:41 PM

What's their idea of public

kian•6/29/22, 10:41 PM

used-in-prod-but-undocumented or no beta APIs haha

kian•6/29/22, 10:42 PM

kian•6/29/22, 10:42 PM

kian•6/29/22, 10:42 PM

Rules out a lotta APIs

Cyb3r-Jak3•6/29/22, 10:42 PM

we'll need to wait for the public API documentation to land to add support here but it's fine to leave the PR pending that for now.

https://github.com/cloudflare/cloudflare-go/pull/936#issuecomment-1159904799

Kkian Rules out a lotta APIs

Cyb3r-Jak3•6/29/22, 10:42 PM

This is true

beelzabub•6/29/22, 10:42 PM

Naw. If you put up a PR I can work with Jacob. We're

beelzabub•6/29/22, 10:43 PM

As long as it's create/delete bucket + create/delete R2 API token

beelzabub•6/29/22, 10:43 PM

Let me double-check with my EM first though

kian•6/29/22, 10:43 PM

I guess it'll never change too wildly from this PUT https://api.cloudflare.com/client/v4/accounts/:account_id/r2/buckets/:bucket_namePUT https://api.cloudflare.com/client/v4/accounts/:account_id/r2/buckets/:bucket_name

kian•6/29/22, 10:44 PM

Currently has no body though - I guess there might be one when storage tiers are a thing?

kian•6/29/22, 10:44 PM

However far into the future that is

beelzabub•6/29/22, 10:45 PM

I think just send a {}{} as the body. We'll ignore it on the R2 side and it'll future proof us for later.

Bbeelzabub Let me double-check with my EM first though

Cyb3r-Jak3•6/29/22, 10:45 PM

Happy to put in the PRs I just don't want to get rejected

beelzabub•6/29/22, 10:48 PM

Is there a reason you can't use the s3 terraform provider?

beelzabub•6/29/22, 10:50 PM

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/custom-service-endpoints

Terraform Registry

Quick question, so what do you think about using R2 for an Unreal Engine S3 Shared Derived Data Cach

Similar Threads

Similar Threads

Similar Threads