R2 is just the backend storage, so the solution is to get your site to a place where it can accept u

R2 is just the backend storage, so the solution is to get your site to a place where it can accept uploads >100MB. Probably the easiest way to do that will be to disable Cloudflare proxy.

Again, using (or not using) R2 doesn’t change your site’s upload limitation. Disable R2 and you’d still have the same upload limitation you are seeing.

Ddigitalpoint R2 is just the backend storage, so the solution is to get your site to a place w...

maco6517•8/22/24, 1:12 PM

ok thanks sir

Barry_Based_Benson•8/22/24, 1:25 PM

@yuvraj are you actually sending a plaintext file or video/mp4 in that case?

BBarry_Based_Benson @yuvraj are you actually sending a plaintext file or video/mp4 in that case?

yuvraj•8/22/24, 1:35 PM

a video/mp4, with content type plaintext in the headers
If i remove the md5 hash/content size inside putobjectcommand, I think I can send a txt file, even though I set contentype to video/mp4 in the putobjectcommand

yuvraj•8/22/24, 1:36 PM

my current conclusion is ContentType inside PutObjectCommand does nothing for presigned urls

Barry_Based_Benson•8/22/24, 1:39 PM

Try set content type to video/mp4 when generating, and actually send a whole plain text file, ignoring what you set in the headers before using the PutObject. Unless that's what you're already doing and I misunderstood

yuvraj•8/22/24, 1:39 PM

ok i'll test with that

yuvraj•8/22/24, 1:45 PM

I generated a presigned url with contenttype to "video/mp4" in putobjectcommand
I uploaded a json file using that url in postman & it worked

yuvraj•8/22/24, 1:46 PM

Barry_Based_Benson•8/22/24, 1:46 PM

I think md5 + size would usually be enough of a validation to protect you anyways

yuvraj•8/22/24, 1:47 PM

yep I'm using those
but the contenttype does nothing xd

Barry_Based_Benson•8/22/24, 1:48 PM

Are you executing this file after on your server, or just storing it?

Barry_Based_Benson•8/22/24, 1:49 PM

Because in that case files can contain malware anyways, even if they're a pdf or video

Barry_Based_Benson•8/22/24, 1:50 PM

Even if they might be the type you expect

Original message was deleted

yuvraj•8/22/24, 2:38 PM

yeah serverside is the best solution for max security, but bandwidth might be an issue
md5 is done on the client, so can be tampered as you mentioned

BBarry_Based_Benson Are you executing this file after on your server, or just storing it?

yuvraj•8/22/24, 2:42 PM

just storing files, i'm not really worried about malwares
i just don't want users to store huge files and abuse my endpoint haha
1st time learning about file storage, just want to make sure I'm doing the best practices

yuvraj•8/22/24, 2:43 PM

thank you all for you help, I appreciate it

Barry_Based_Benson•8/22/24, 4:07 PM

@yuvraj In that case the md5 + size validation should be good enough. If a user spoofs the type then that'll affect them if you serve it back to them for viewing/download on a client-side, but at that point it would be their fault and not your responsibility, as long as it doesn't breach the security of your server/other users then it's not really an issue you need to worry about.

Barry_Based_Benson•8/22/24, 4:08 PM

And yeah presigned urls in that case are likely still your best bet since processing the files on the server itself can be heavy on memory/server storage, even if you stream

Barry_Based_Benson•8/22/24, 4:12 PM

People who want their stuff to scale well should

Barry_Based_Benson•8/22/24, 4:18 PM

It's not just scale either, it's probably faster to upload with a PutObject url as you don't need to send the file to the server and THEN the bucket, instead sending the file straight to the bucket

Barry_Based_Benson•8/22/24, 4:19 PM

Sometimes it's better to sacrifice reliability. Don't cater to spoofers if they only affect themselves lol

Barry_Based_Benson•8/22/24, 4:21 PM

@Space Wdym? If its not gonna affect storage space or security or anything else that matters then don't worry about it. If a user spoofs then they should know what they're getting themselves into. Like you wouldn't add extra code to the client side just in case someone uses inspect element would you?

Barry_Based_Benson•8/22/24, 4:22 PM

As you should

Barry_Based_Benson•8/22/24, 4:23 PM

But if someone makes a dumb suggestion and gets a dumb response then what did they expect. As long as it doesn't affect anyone else

Original message was deleted

Barry_Based_Benson•8/22/24, 4:24 PM

What checks?

Barry_Based_Benson•8/22/24, 4:24 PM

I disagree

Barry_Based_Benson•8/22/24, 4:24 PM

That's what generating the url on the server with the correct headers are for

Barry_Based_Benson•8/22/24, 4:26 PM

The server? Since when is a user gonna access the server. If that's the case then you have bigger issues than just making sure the file is correct

Barry_Based_Benson•8/22/24, 4:27 PM

The url generation doesn't touch the client-side

Barry_Based_Benson•8/22/24, 4:30 PM

The needed metadata is sent to the server, the server itself validates the metadata e.g. size and then sets it as a necessary header on the url, send the url back, and then the client-side uses the url to put the file. If the client lies when using the url (e.g. mismatching what they sent to the server in terms of metadata like size) then the endpoint that's connected to the presigned url will reject it. This rejection isn't based on a client sending the header, instead it uses the file itself that got uploaded to the put url.

digitalpointOP•8/22/24, 4:32 PM

Any file upload mechanism is “client authoritative” because the file being uploaded is from the client.

digitalpointOP•8/22/24, 4:33 PM

You can still validate the uploaded file after the fact and delete it if it’s not what’s expected (same as if you weren’t using R2).

digitalpointOP•8/22/24, 4:35 PM

Well there’s only so much you can do to pre-validate something being uploaded by a client. You can enforce certain rules but the client can always “lie”… so things should still be validated after upload. This is not an R2 issue though. That’s the same for anything/any mechanism you use for user uploads.

Barry_Based_Benson•8/22/24, 4:35 PM

Balancing resource usage and reliability is important. If all companies think that way, with a who-knows-how-many validations for attributes that, if they were wrong, wouldn't even affect anyone other than the user, then those companies might not scale super well

Barry_Based_Benson•8/22/24, 4:36 PM

@Space

Barry_Based_Benson•8/22/24, 4:36 PM

Because its a pretty important thing?

Barry_Based_Benson•8/22/24, 4:36 PM

For a service that wants to expand

Barry_Based_Benson•8/22/24, 4:37 PM

And like I said before, it's not just scale, UX is affected that way because processing files on the server affects speed

digitalpointOP•8/22/24, 4:38 PM

User generated content should be validated regardless of the scale or storage system. If a user is supposed to upload an image, make sure it’s actually an image when you receive it for example. Same if you are just uploading direct to a single origin server (not using R2). A need to trust users for anything is already fail model for anything.

Barry_Based_Benson•8/22/24, 4:43 PM

@digitalpoint validating everything can still take a lot of resources, you sacrifice reliability slightly with the presigned urls but if a user can't spoof anything that affect the server negatively nor other users (aka only affecting themselves) then it's not really an issue to waste time on. A user that tampers with things like that should not expect a good experience on the client-side. Of course if you can validate unimportant attributes without significant resource usage then go ahead, but something like content type isn't as easy to validate. And even if you do, how do you know that's correct? Does your algorithm scan the entire file for the type, or just the magic numbers at the start? Well what if the user spoofs that too?

Ddigitalpoint User generated content should be validated regardless of the scale or storage sy...

Chaika•8/22/24, 4:46 PM

I think it's way more complex then that. A lot of companies use S3 presigned for uploads on websites, it really depends on your scope (is this just a generic upload website), or is this an internal company thing so if an admin uploads an img it's whatever, or is this an image all users would see?

CChaika I think it's way more complex then that. A lot of companies use S3 presigned for...

Barry_Based_Benson•8/22/24, 4:46 PM

Exactly

Barry_Based_Benson•8/22/24, 4:46 PM

Depends on who it affects

Chaika•8/22/24, 4:47 PM

Images are small enough that you may want to just proxy them through your backend and check, for larger things makes way less sense for it all go through your backend

digitalpointOP•8/22/24, 4:48 PM

I guess it just depends on the content and the site can make that decision. For me, I don’t even trust admins to do things right. Like images they upload have EXIF data stripped and converted to WEBP when it’s uploaded.

digitalpointOP•8/22/24, 4:48 PM

And as part of that process, it’s easy enough to check if the input is a valid image since it’s being mucked with anyway.

digitalpointOP•8/22/24, 4:50 PM

Same with user uploaded .zip files… if the files are supposed to be in a specific hierarchy in the archive, check that. Obviously you can’t check everything, but a lot you can.

Ddigitalpoint Same with user uploaded .zip files… if the files are supposed to be in a specifi...

Chaika•8/22/24, 4:52 PM

yea, sometimes can do it client side too. I have something where users pick images, the client-side code checks image, and then they can pick to upload directly to bucket. It's b2b software so I really doubt someone would try to jump the check but even if they do it's on them, only they can see their own messed up imgs lol, just want to have sane max upload limits so they can't say "this 1gb file is totally an image"

Barry_Based_Benson•8/22/24, 4:54 PM

And presigned urls make size validation easy

Chaika•8/22/24, 4:56 PM

well you clamp that down to reasonable limits lol

R2 is just the backend storage, so the solution is to get your site to a place where it can accept u

Similar Threads

Similar Threads

Similar Threads