Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare address, but I see t

Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare address, but I see the R2 devs are very active here so might be a better place to ask that part.

We're currently working on our stable provider/module registry, really mostly the JSON serving layer (not the artifacts). It seems like R2 would basically be perfect for that. However, it seems like by default R2 is single-region. Would you be able to make a recommendation on how to host a bunch of json files (1.6GB of json spread across a couple tens of thousands of files) in a multi-region way behind a custom domain?

I suppose it would be multiple single-region R2 buckets frontend by a CDN / Worker (and we'd just rclone the jsons to all the buckets), but I'm far from an expert at the CloudFlare offering and can't find much info online about this.

We have a public rfc up for this, fwiw, which contains a bit more background: https://github.com/opentofu/opentofu/issues/841

GitHub

[RFC] Pre-generating OpenTofu Registry v1 API Responses · Issue #84...

Introduction Note This proposal is part of the Homebrew-like Registry Design. This RFC introduces a strategy for pre-generating API responses for the OpenTofu registry using predefined JSON files a...

cube2222OP•11/9/23, 1:39 PM

(or I suppose another question - is the R2 "Europe" region really single-region, or is it replicated across a few geographically distributed regions across Europe)

Ccube2222 (or I suppose another question - is the R2 "Europe" region really single-region,...

Erisa•11/9/23, 1:41 PM

The latter, each region covers a broad area and data is distributed within that region - but thats intended for availability, and doesn't impact performance.
This old message is still true r2

cube2222OP•11/9/23, 1:42 PM

Awesome, thanks for the quick reponse @Erisa | Support Engineer !

Ccube2222 Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare ...

Erisa•11/9/23, 1:45 PM

Regarding

Would you be able to make a recommendation on how to host a bunch of json files (1.6GB of json spread across a couple tens of thousands of files) in a multi-region way behind a custom domain?

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

I will also note though if your data is small enough (<25MB per file) and you are okay with writing a Worker, you may have better performance with KV

Erisa•11/9/23, 1:45 PM

Though... uploading the data to KV would not be as simple as R2

Erisa•11/9/23, 1:45 PM

So its a game of tradeoffs

Erisa•11/9/23, 1:46 PM

There is some guidance here on choosing a storage product: https://developers.cloudflare.com/workers/learning/storage-options/

Storage options guide · Cloudflare Workers docs

Storage options available on Cloudflare's developer platform that can be used with Workers.

cube2222OP•11/9/23, 1:54 PM

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

That's very useful information!

cube2222OP•11/9/23, 1:55 PM

Though... uploading the data to KV would not be as simple as R2
So its a game of tradeoffs

Makes sense. We're not too latency sensitive here, so I think we'll start with a public R2, and if we need to, we can always improve on that.

cube2222OP•11/9/23, 1:55 PM

Thanks a lot for the help!

Erisa•11/9/23, 1:56 PM

No problem, hope it goes well and let us know if you have more questions

sin•11/9/23, 4:46 PM

Does anyone know the expiration time of S3 Presigned URLs? If the client has not completed the upload before the expiration, will the connection be interrupted? Or does this refer only to the start time of the request?

Ssin Does anyone know the expiration time of S3 Presigned URLs? If the client has not...

Erisa•11/9/23, 5:07 PM

Start time, ongoing connections are not interrupted and complete like normal.
The same is true for downloads; you can start downloading a file from presigned url and will keep downloading even if it expires halfway

Ccake We are seeing that R2 API tokens / secrets are expiring, and we're getting `aws....

Sid•11/9/23, 5:26 PM

That's to be expected no? Your token expired so API access was revoked? I may be misunderstanding

KKentaro [R2 API Token] I received "Access Denied" for S3 API when I use "Object Read & W...

Sid•11/9/23, 5:27 PM

What action are you issuing? Object tokens will not allow bucket-level configuration actions

nitsky•11/9/23, 5:28 PM

Hi, I have a REST API where a subset of the routes return objects from R2 which can be made publicly available. Rather than reading the objects from my API to serve those requests, I would like to serve them directly from Cloudflare with the public bucket feature. However, I still want all other routes to hit my API servers. Is this possible? Thanks!

Nnitsky Hi, I have a REST API where a subset of the routes return objects from R2 which ...

Sid•11/9/23, 5:31 PM

Note that making a bucket public by attaching a custom domain will make the entire bucket public. There currently isn't a way to keep certain prefixes private.
I'd recommend putting up a Worker (or using your existing server) in front of your bucket to decide if you want to serve a certain object

SSid Note that making a bucket public by attaching a custom domain will make the enti...

nitsky•11/9/23, 5:32 PM

In my case, everything can be public! I am hoping to take advantage of that to reduce costs.

Nnitsky In my case, everything can be public! I am hoping to take advantage of that to r...

Sid•11/9/23, 5:33 PM

Ah, in that case, have you looked into Page Rules on your domain?

Original message was deleted

Chaika•11/9/23, 7:37 PM

This is the channel for Cloudflare's R2 Object Storage. If you have a question about AWS billing, you should probably reach out to AWS, not much we can do here.

Sshep what are the rate limits on r2?

WakaTime•11/9/23, 9:06 PM

did you ever find an answer? I'm wondering the same about R2 max requests per second

shep•11/9/23, 9:06 PM

yeah

shep•11/9/23, 9:06 PM

theres no rate limit

shep•11/9/23, 9:06 PM

well i didnt hit any

shep•11/9/23, 9:06 PM

so was doing 10k batches

WakaTime•11/9/23, 9:07 PM

what avg req/sec did you make?

WakaTime•11/9/23, 9:07 PM

those were writes?

WakaTime•11/9/23, 9:10 PM

1k/sec reads would be enough for me, as long as latency is low. i'll just try it out and see compared to s3

shep•11/9/23, 9:10 PM

it was 10k a sec

Unsmart•11/9/23, 9:10 PM

There are no rate limits on R2. But you can break the service if you are in the range of like ~5-10k RPS potentially

shep•11/9/23, 9:10 PM

well this was just for migration to r2

shep•11/9/23, 9:11 PM

but im prob doing like 10k /10sec class b

shep•11/9/23, 9:11 PM

the hotlink protection is good also

Sshep the hotlink protection is good also

WakaTime•11/9/23, 9:16 PM

oh, these files are only accessed via API not public

shep•11/9/23, 9:21 PM

yeah then what the mod said

Sshep it was 10k a sec

Zora•11/9/23, 11:16 PM

That's very nice, for comparison backblaze b2 has a request rate limit of double digits before it starts giving 429 errors

Zora•11/9/23, 11:16 PM

I learned that the hard way

kian•11/10/23, 12:04 AM

Same as gzip.

kian•11/10/23, 12:05 AM

Cloudflare will decompress it (if necessary) based on the client's Accept-Encoding header

snow21•11/10/23, 12:08 AM

Is anyone using R2 storage with Flutter or Dart?

iot•11/10/23, 5:20 AM

i uses R2 for storing media files for my production server (django rest API's ), mainly i do converting all types of files to pdf and saving in R2 storage, while converting jpg, png theree is no problem with that , but when i do for MS files (doc, docx, xls, xlsx, ppt, pptx) , it gives me problem.

I think there is issue with output file url ( as it seems path containg X-Amz-Algorithm=, X-Amz-Credential=, X-Amz-Expires=, X-Amz-SignedHeaders=host&X-Amz-Signature=)etc, please give me the solution, what should i do for that?

ZZora That's very nice, for comparison backblaze b2 has a request rate limit of double...

WakaTime•11/10/23, 8:50 AM

DigitalOcean Spaces has a limit of 500 req/sec per bucket and 1500 req/sec for all buckets.. so only R2 and S3 can compete on request rates

Migmac•11/10/23, 9:25 AM

Question: Is there a way I can generate a presigned url with a custom Content-Disposition header?

Something like this?

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

kian•11/10/23, 9:27 AM

Provide it in the headers for RequestInit.

kian•11/10/23, 9:28 AM

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

kian•11/10/23, 9:29 AM

you'd probably want to add

allHeaders: true

allHeaders: true

into the

aws

aws

block as well, next to

signQuery: true

signQuery: true

kian•11/10/23, 9:29 AM

that only signs it though - the client still has to provide it in the request

const r2 = new AwsClient({ accessKeyId: env.CF_ACCESS_KEY_ID, secretAccessKey: env.CF_ACCESS_KEY_SECRET, }) const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`) url.pathname = obj.path url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`) const _signed = await r2.sign(new Request(url, { method: 'GET', contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this? }), { aws: { signQuery: true } }) const signed = _signed.url

Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare address, but I see t

Similar Threads

Similar Threads

Similar Threads