So, objects may be cached and writing to R2 won't purge that

Kkian So, objects may be cached and writing to R2 won't purge that

E

Ercan•1/21/24, 8:52 PM

Hm, but if an object from r2 is already not in the cache and just uploaded, and if I try to access it using custom domain, why would it fail if it is already not in the cache, and it should retrieve it from the source right? Maybe I am wrong, can you explain a bit more

K

kianOP•1/21/24, 8:53 PM

404's are cached but it's impossible to tell why it happened without a repro

Kkian 404's are cached but it's impossible to tell why it happened without a repro

E

Ercan•1/21/24, 8:54 PM

I see, I will try to get the error and time, if I see one more time, so we could investigate maybe.

Kkian 404's are cached but it's impossible to tell why it happened without a repro

E

Ercan•1/21/24, 8:56 PM

I also wanna mention what I am planning for 100% relaibility. I will have a custom image resizer server running that will handle resizing etc, and I will source it to directly r2 without custom domain. But I will connect my own cloudlfare domain to my image resizer, so the results of my image resizer would be cached. And I am hoping I can reach the maximum reliability with this setup.

P

pinky•1/21/24, 9:21 PM

Hello, when I have a private bucket and allow access via Worker with the following code, is it going to load the file into Worker's memory and is there any usage impact compared to accessing a file in a public bucket?

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

K

kianOP•1/21/24, 9:21 PM

Nope

K

kianOP•1/21/24, 9:22 PM

Public buckets are better if you're not adding auth though

P

pinky•1/21/24, 9:26 PM

Is it also the case with the upload? Because I see various ways to upload a file when using worker API.

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

Just to confirm, the first approach is also memory free and almost not blocking the worker but the second approach needs to load everything?

K

kianOP•1/21/24, 9:38 PM

Yes

K

kianOP•1/21/24, 9:39 PM

FormData can't be uploaded as-is since it isn't just the file, it has other things

C

chip•1/22/24, 1:20 AM

Hey! Is it possible to slap on

x-amz-expiration

x-amz-expiration

x-amz-expiration

x-amz-expiration in metadata for objects to delete them, without having set up a lifecycle in r2?

H

hoalongnatsu•1/22/24, 1:45 AM

I added my credit card and active R2 subscription, but when I go to R2 page then it shows like the image below. I press "Add R2 subscription to my account account" many times but it won't work

.

H

hoalongnatsu•1/22/24, 1:47 AM

How can I work with this problem?

Cchip Hey! Is it possible to slap on `x-amz-expiration` in metadata for objects to del...

K

kianOP•1/22/24, 1:49 AM

Nope, but I also don't think S3 supports that either?

Cchip Hey! Is it possible to slap on `x-amz-expiration` in metadata for objects to del...

K

Karew•1/22/24, 1:49 AM

No, only lifecycles are supported at the moment

C

chip•1/22/24, 1:55 AM

Sounds good! Thanks!

K

kianOP•1/22/24, 3:33 AM

People have had issues with Minio’s JS SDK before when using bucket-scoped tokens so it probably isn’t an issue with your code

C

constant1894•1/22/24, 1:04 PM

Hi Everyone! I'm struggeling to set the content type of a .m3u8 playlist. I can see the correct content type does reach the worker, but once the file is stored on R2 the type shows -- and the file cannot be played. Is it possible that my m3u8 file has some issues?

Cconstant1894 Hi Everyone! I'm struggeling to set the content type of a .m3u8 playlist. I can ...

H

Hello, I’m Allie!•1/22/24, 1:05 PM

Are you adding a content-type with

R2PutOptions

R2PutOptions

R2PutOptions

R2PutOptions? https://developers.cloudflare.com/r2/api/workers/workers-api-reference/#r2putoptions

Workers API reference · Cloudflare R2 docs

The in-Worker R2 API is accessed by binding an R2 bucket to a Worker. The Worker you write can expose external access to buckets via a route or …

HHello, I’m Allie!Are you adding a content-type with `R2PutOptions`? https://developers.cloudflare...

C

constant1894•1/22/24, 1:06 PM

Mmm I'm only passing the request.body to the put function via the worker

Cconstant1894 Mmm I'm only passing the request.body to the put function via the worker

H

Hello, I’m Allie!•1/22/24, 1:06 PM

Yeah, so adding the content-type there might help

HHello, I’m Allie!Yeah, so adding the content-type there might help

C

constant1894•1/22/24, 1:07 PM

I'll give it a go, but it's weird because all my other content types work.

Original message was deleted

H

Hello, I’m Allie!•1/22/24, 1:08 PM

If it is cached, no. Otherwise, yes

C

constant1894•1/22/24, 1:09 PM

@HardlyWorkin' Sorry for the dumb question, if my current worker api looks like this: await env.Testing_Bucket.put(key, request.body); How would i add the content type?

Cconstant1894 @HardlyWorkin' Sorry for the dumb question, if my current worker api looks like ...

H

Hello, I’m Allie!•1/22/24, 1:11 PM

await env.Testing_bucket.put(key, request.body, request.headers);

await env.Testing_bucket.put(key, request.body, request.headers);

await env.Testing_bucket.put(key, request.body, request.headers);

await env.Testing_bucket.put(key, request.body, request.headers);

should be fine

Original message was deleted

H

Hello, I’m Allie!•1/22/24, 1:12 PM

Yes

Original message was deleted

H

Hello, I’m Allie!•1/22/24, 1:12 PM

Should definitely be doable. Mine is slow, but it works: https://maps.goalastair.com

H

Hello, I’m Allie!•1/22/24, 1:12 PM

Based on Protomaps

HHello, I’m Allie!```ts await env.Testing_bucket.put(key, request.body, request.headers); ```shoul...

C

constant1894•1/22/24, 1:13 PM

Thanks, just tested in, but still getting the same issue as before.

S

Shubham•1/22/24, 1:43 PM

We are facing some issue while uploading file larger than 5Gb in the R2. We have migrated to R2 and we started facing issue, we cannot ask client to update their code to use multipart upload, also multipart upload is not that straight forward. We have already submitted the request from https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVOboH9SlutMonzbhCxuuuOmiU1L_I5O2CFbXf_XXMRg/viewform?pli=1&pli=1 but no reply yet. We are facing too much issue due to this. We have use case of upload upto 15Gb. Could anybody from cloudflare team can help on this?

@Skye @Matt

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

SShubham We are facing some issue while uploading file larger than 5Gb in the R2. We have...

S

Shubham•1/22/24, 1:56 PM

Account ID: 83025b28472d6aa2bf5ae59f3724aa78

E

Erisa•1/22/24, 2:15 PM

Please don't ping people for this. If you need urgent help you will need to open a support ticket, Discord server assistance is community supported and best effort.
To my knowledge 5GB is a fixed limit that should not be too unusual for an S3-compatible service and S3 multipart uploads are effectively an industry standard when handling large uploads.

S

Shubham•1/22/24, 2:49 PM

@Erisa | Support Engineer I raised the request using the request form and I tried opening support ticket but it is redirecting to discord for any R2 related issue. I understand that but we have migrated around 1TB of data to R2 and we are facing now issue which we were not facing with other provider. It was just a humble request to look into the request raised by Form as the business is getting impacted.

S

Shubham•1/22/24, 2:55 PM

@Space - Ping in replies I am asking because there was option for that request, we were using already another provider where we didn't had issue and after migration it is very difficult to move back more that 1TB data. If it cannot be done then we will be stuck with multi part option only that I am aware.

U

Unsmart•1/22/24, 2:57 PM

Workers would have the same 5GB limit. Probably lower if they aren’t enterprise

U

Unsmart•1/22/24, 2:58 PM

Yes. The 5GB limit is the max you can proxy through cloudflare on ent

S

Shubham•1/22/24, 2:58 PM

Exactly, before that we were streaming data with 7-8 GB

S

Shubham•1/22/24, 2:59 PM

If this request can be done for increase we will be happy otherwise we need to involve many stakeholders. But Thank you anyways

U

Unsmart•1/22/24, 2:59 PM

5GB is the absolute max you have to use multipart

UUnsmart Yes. The 5GB limit is the max you can proxy through cloudflare on ent

E

Erisa•1/22/24, 3:22 PM

Proxied upload can be increased above 5GB on Ent (and the max without contacting support/success is 500MB), but you start running into tradeoffs and needing to disable body buffering when you get that high so its usually not worth it.

S

Scruffy•1/22/24, 5:36 PM

I've got a corrupted file upload issue. On the web UI the files are listed with their respective upload dates and sizes. When I open a specific object in the web UI it shows as having 0 Bytes in size and any attempt to access it via the public URL returns a 404

S

Scruffy•1/22/24, 5:39 PM

S

Scruffy•1/22/24, 5:40 PM

To be clear my bucket does have public access enabled and it works for all my other objects

SScruffy I've got a corrupted file upload issue. On the web UI the files are listed with ...

M

Matt Silverlock•1/22/24, 5:42 PM

Can you please open a Support ticket, including the account ID and bucket ID? That's the best place for us to investigate this — any suggestion of "corruption" is taken very seriously

MMatt Silverlock Can you please open a Support ticket, including the account ID and bucket ID? Th...

S

Scruffy•1/22/24, 5:42 PM

Sure i dont actually know how to open a support ticket

U

Unsmart•1/22/24, 5:42 PM

The issue is the hashtag... when accessing by url you need to percent encode

%23

%23

%23

%23

U

Unsmart•1/22/24, 5:43 PM

https://support.google.com/richmedia/answer/190941?hl=en

UUnsmart <https://support.google.com/richmedia/answer/190941?hl=en>

S

Scruffy•1/22/24, 5:43 PM

yep thats correct

S

Scruffy•1/22/24, 5:43 PM

ty

So, objects may be cached and writing to R2 won't purge that

Similar Threads

Similar Threads

Similar Threads