(or I suppose another question - is the R2 "Europe" region really single-region, or is it replicated

Ccube2222 (or I suppose another question - is the R2 "Europe" region really single-region,...

E

Erisa•11/9/23, 1:41 PM

The latter, each region covers a broad area and data is distributed within that region - but thats intended for availability, and doesn't impact performance.
This old message is still true r2

C

cube2222OP•11/9/23, 1:42 PM

Awesome, thanks for the quick reponse @Erisa | Support Engineer !

Ccube2222 Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare ...

E

Erisa•11/9/23, 1:45 PM

Regarding

Would you be able to make a recommendation on how to host a bunch of json files (1.6GB of json spread across a couple tens of thousands of files) in a multi-region way behind a custom domain?

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

I will also note though if your data is small enough (<25MB per file) and you are okay with writing a Worker, you may have better performance with KV

E

Erisa•11/9/23, 1:45 PM

Though... uploading the data to KV would not be as simple as R2

E

Erisa•11/9/23, 1:45 PM

So its a game of tradeoffs

E

Erisa•11/9/23, 1:46 PM

There is some guidance here on choosing a storage product: https://developers.cloudflare.com/workers/learning/storage-options/

Storage options guide · Cloudflare Workers docs

Storage options available on Cloudflare's developer platform that can be used with Workers.

C

cube2222OP•11/9/23, 1:54 PM

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

That's very useful information!

C

cube2222OP•11/9/23, 1:55 PM

Though... uploading the data to KV would not be as simple as R2
So its a game of tradeoffs

Makes sense. We're not too latency sensitive here, so I think we'll start with a public R2, and if we need to, we can always improve on that.

C

cube2222OP•11/9/23, 1:55 PM

Thanks a lot for the help!

E

Erisa•11/9/23, 1:56 PM

No problem, hope it goes well and let us know if you have more questions

S

sin•11/9/23, 4:46 PM

Does anyone know the expiration time of S3 Presigned URLs? If the client has not completed the upload before the expiration, will the connection be interrupted? Or does this refer only to the start time of the request?

Ssin Does anyone know the expiration time of S3 Presigned URLs? If the client has not...

E

Erisa•11/9/23, 5:07 PM

Start time, ongoing connections are not interrupted and complete like normal.
The same is true for downloads; you can start downloading a file from presigned url and will keep downloading even if it expires halfway

Ccake We are seeing that R2 API tokens / secrets are expiring, and we're getting `aws....

S

Sid•11/9/23, 5:26 PM

That's to be expected no? Your token expired so API access was revoked? I may be misunderstanding

KKentaro [R2 API Token] I received "Access Denied" for S3 API when I use "Object Read & W...

S

Sid•11/9/23, 5:27 PM

What action are you issuing? Object tokens will not allow bucket-level configuration actions

N

nitsky•11/9/23, 5:28 PM

Hi, I have a REST API where a subset of the routes return objects from R2 which can be made publicly available. Rather than reading the objects from my API to serve those requests, I would like to serve them directly from Cloudflare with the public bucket feature. However, I still want all other routes to hit my API servers. Is this possible? Thanks!

Nnitsky Hi, I have a REST API where a subset of the routes return objects from R2 which ...

S

Sid•11/9/23, 5:31 PM

Note that making a bucket public by attaching a custom domain will make the entire bucket public. There currently isn't a way to keep certain prefixes private.
I'd recommend putting up a Worker (or using your existing server) in front of your bucket to decide if you want to serve a certain object

SSid Note that making a bucket public by attaching a custom domain will make the enti...

N

nitsky•11/9/23, 5:32 PM

In my case, everything can be public! I am hoping to take advantage of that to reduce costs.

Nnitsky In my case, everything can be public! I am hoping to take advantage of that to r...

S

Sid•11/9/23, 5:33 PM

Ah, in that case, have you looked into Page Rules on your domain?

Original message was deleted

C

Chaika•11/9/23, 7:37 PM

This is the channel for Cloudflare's R2 Object Storage. If you have a question about AWS billing, you should probably reach out to AWS, not much we can do here.

Sshep what are the rate limits on r2?

W

WakaTime•11/9/23, 9:06 PM

did you ever find an answer? I'm wondering the same about R2 max requests per second

S

shep•11/9/23, 9:06 PM

yeah

S

shep•11/9/23, 9:06 PM

theres no rate limit

S

shep•11/9/23, 9:06 PM

well i didnt hit any

S

shep•11/9/23, 9:06 PM

so was doing 10k batches

W

WakaTime•11/9/23, 9:07 PM

what avg req/sec did you make?

W

WakaTime•11/9/23, 9:07 PM

those were writes?

W

WakaTime•11/9/23, 9:10 PM

1k/sec reads would be enough for me, as long as latency is low. i'll just try it out and see compared to s3

S

shep•11/9/23, 9:10 PM

it was 10k a sec

U

Unsmart•11/9/23, 9:10 PM

There are no rate limits on R2. But you can break the service if you are in the range of like ~5-10k RPS potentially

S

shep•11/9/23, 9:10 PM

well this was just for migration to r2

S

shep•11/9/23, 9:11 PM

but im prob doing like 10k /10sec class b

S

shep•11/9/23, 9:11 PM

the hotlink protection is good also

Sshep the hotlink protection is good also

W

WakaTime•11/9/23, 9:16 PM

oh, these files are only accessed via API not public

S

shep•11/9/23, 9:21 PM

yeah then what the mod said

Sshep it was 10k a sec

Z

Zora•11/9/23, 11:16 PM

That's very nice, for comparison backblaze b2 has a request rate limit of double digits before it starts giving 429 errors

Z

Zora•11/9/23, 11:16 PM

I learned that the hard way

K

kian•11/10/23, 12:04 AM

Same as gzip.

K

kian•11/10/23, 12:05 AM

Cloudflare will decompress it (if necessary) based on the client's Accept-Encoding header

S

snow21•11/10/23, 12:08 AM

Is anyone using R2 storage with Flutter or Dart?

I

iot•11/10/23, 5:20 AM

i uses R2 for storing media files for my production server (django rest API's ), mainly i do converting all types of files to pdf and saving in R2 storage, while converting jpg, png theree is no problem with that , but when i do for MS files (doc, docx, xls, xlsx, ppt, pptx) , it gives me problem.

I think there is issue with output file url ( as it seems path containg X-Amz-Algorithm=, X-Amz-Credential=, X-Amz-Expires=, X-Amz-SignedHeaders=host&X-Amz-Signature=)etc, please give me the solution, what should i do for that?

ZZora That's very nice, for comparison backblaze b2 has a request rate limit of double...

W

WakaTime•11/10/23, 8:50 AM

DigitalOcean Spaces has a limit of 500 req/sec per bucket and 1500 req/sec for all buckets.. so only R2 and S3 can compete on request rates

M

Migmac•11/10/23, 9:25 AM

Question: Is there a way I can generate a presigned url with a custom Content-Disposition header?

Something like this?

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

const r2 = new AwsClient({
  accessKeyId: env.CF_ACCESS_KEY_ID,
  secretAccessKey: env.CF_ACCESS_KEY_SECRET,
})

const url = new URL(`https://${env.BUCKET_NAME}.${env.CF_ACCOUNT_ID}.r2.cloudflarestorage.com`)

url.pathname = obj.path
url.searchParams.set('X-Amz-Expires', `${60 * 60 * 24 * env.STRUCTURE_EXPIRY}`)

const _signed = await r2.sign(new Request(url, { 
  method: 'GET',
  contentDisposition: `attachment; filename="${obj.name}"`, // <------------------------ Something like this?
}), { aws: { signQuery: true } })
const signed = _signed.url

K

kian•11/10/23, 9:27 AM

Provide it in the headers for RequestInit.

K

kian•11/10/23, 9:28 AM

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

new Request(url, {
  headers: {
    'content-disposition': `attachment; filename="${obj.name}"`
  }
})

K

kian•11/10/23, 9:29 AM

you'd probably want to add

allHeaders: true

allHeaders: true

allHeaders: true

allHeaders: true into the

aws

aws

aws

aws block as well, next to

signQuery: true

signQuery: true

signQuery: true

signQuery: true

K

kian•11/10/23, 9:29 AM

that only signs it though - the client still has to provide it in the request

M

Migmac•11/10/23, 9:43 AM

Awesome! Thank you @kian

=]

=]

=]

=]

(or I suppose another question - is the R2 "Europe" region really single-region, or is it replicated

Similar Threads

Similar Threads

Similar Threads