I'd assume its not there because its not a limit they impose directly its something that number of r

I'd assume its not there because its not a limit they impose directly its something that number of requests and concurrency can eventually break the system

UUnsmart I'd assume its not there because its not a limit they impose directly its someth...

dave•1/26/24, 7:29 PM

but doing like 1,000 concurrent puts on the same bucket seems like it'll always fail, no?

UnsmartOP•1/26/24, 7:33 PM

Always maybe right now but in the future not really. It might depend on size of files not sure

Original message was deleted

dave•1/27/24, 1:27 AM

Not yet. You could do a cron Worker that lists new files in a bucket and tracks it, but kinda a pain atm.

Original message was deleted

Hello, I’m Allie!•1/27/24, 12:57 PM

If you run all the uploads through a Worker, then you could just call a function there

Original message was deleted

Hello, I’m Allie!•1/27/24, 1:11 PM

Yes, it would just require a little bit of work

Original message was deleted

dave•1/27/24, 3:03 PM

You will be limited to 100MB on uploads to your Worker if you’re on the free or Pro plan.

Original message was deleted

dave•1/27/24, 3:04 PM

You could look into using gzip on your data, might save you a lot. I know for us, gzip saved us a ton on storing JSON in buckets.

Original message was deleted

dave•1/27/24, 3:05 PM

Ah, what runtime is doing the compression? Is it a Python script or something?

dave•1/27/24, 3:07 PM

If CPU time isn’t an issue, compressing with lzma should have higher compression.

Original message was deleted

dave•1/27/24, 3:08 PM

why not lz4?

dave•1/27/24, 3:09 PM

oh right I am using zstd for btrfs compression already

dave•1/27/24, 3:12 PM

If the download is going through a Worker, you can decompress it on the fly.

Ddave If the download is going through a Worker, you can decompress it on the fly.

dave•1/27/24, 3:13 PM

Or if it’s being viewed in a browser

Original message was deleted

dave•1/27/24, 3:20 PM

What’s annoying is that the etag won’t be the md5 hash of the entire file if you do chunked uploads.

AAnay-208 So do I’ve to use a worker for it then ?

dave•1/27/24, 4:24 PM

your Worker won't be able to accept POST'd 10GB file

AAnay-208 any alternatives

dave•1/27/24, 4:25 PM

you can sign the Content-LengthContent-Length header, but this requires you to know the size that the user will upload.

AAnay-208 I need a max file size of 5 MB on presigned url. So instead of presigned url, I ...

dave•1/27/24, 4:26 PM

correct.

AAnay-208 I need a max file size of 5 MB on presigned url. So instead of presigned url, I ...

dave•1/27/24, 4:28 PM

you probably also want to use the If-Unmodified-SinceIf-Unmodified-Since header so you don't allow infinite reuploads of the same object.

https://developers.cloudflare.com/r2/api/s3/presigned-urls/#presigned-url-alternative-with-workers

Ddave you can sign the `Content-Length` header, but this requires you to know the size...

Erisa•1/27/24, 4:30 PM

This. The way I would do it would be to have the client send the size of the file in the request that generates the presigned URL, then in your backend code you check that the size is not too large and generate a presigned URL with the appropiate content-length

EErisa This. The way I would do it would be to have the client send the size of the fil...

dave•1/27/24, 4:31 PM

I think the If-Unmodified-SinceIf-Unmodified-Since header is important too, in case R2 adds S3 Versioning support and @Anay - ping in replies enables it without thinking about this edge case

Ddave I think the `If-Unmodified-Since` header is important too, in case R2 adds S3 Ve...

Erisa•1/27/24, 4:32 PM

Yes

AAnay-208 Ya

dave•1/27/24, 4:32 PM

if you wanna get real fancy, you can also sign the user-agentuser-agent header.

AAnay-208 do I add it to request

dave•1/27/24, 4:33 PM

const signed = await r2.sign(
  new Request(url, {
    method: "PUT",
  }),
  {
    aws: { signQuery: true },
    headers: {
      "If-Unmodified-Since": "Tue, 28 Sep 2021 16:00:00 GMT",
      "Content-Length" "1337",
      "User-Agent": "I am legit 1.2.3.4"
    },
  }
);

const signed = await r2.sign(
  new Request(url, {
    method: "PUT",
  }),
  {
    aws: { signQuery: true },
    headers: {
      "If-Unmodified-Since": "Tue, 28 Sep 2021 16:00:00 GMT",
      "Content-Length" "1337",
      "User-Agent": "I am legit 1.2.3.4"
    },
  }
);

dave•1/27/24, 4:36 PM

you could/should also limit the content-type header if you know it ahead of time.

AAnay-208 Do I've to use aws4fetch library, or it can't be used s3-request-presigner

dave•1/27/24, 4:36 PM

oh yeah I'd use aws4fetch

AAnay-208 Do I've to use aws4fetch library, or it can't be used s3-request-presigner

dave•1/27/24, 4:37 PM

for aws4fetch, you need to set allHeaders: trueallHeaders: true so it'll sign all headers, even the "non-standard" ones like User-AgentUser-Agent

dave•1/27/24, 4:38 PM

if you wanna see what special headers S3 can use, see:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_RequestSyntax

PutObject - Amazon Simple Storage Service

Adds an object to a bucket.

dave•1/27/24, 4:39 PM

you could even sign the CookieCookie header

AAnay-208 before returning the presigned uri, the user needs to be authenticated to the ap...

dave•1/27/24, 4:41 PM

do you usually send cookie or an authorization header on requests?

AAnay-208 on the frontend, it gets the cookie, and sends it in the headers on requests.

dave•1/27/24, 4:45 PM

ah hmm now that I think about it, don't sign the cookie header, since presigned URLs aren't gonna be on your domain

AAnay-208 Is it a good idea to set this to If-Unmodified-Since ```js const signed = await ...

dave•1/27/24, 4:54 PM

I think that makes sense.

I would also sign the Content-DispositionContent-Disposition to enforce Content-Disposition: attachmentContent-Disposition: attachment. And yeah, I'd sign User-AgentUser-Agent if it's static for your application.

AAnay-208 And if I’m only accepting only image files any headers to sign?

dave•1/27/24, 4:55 PM

do you know what image type?

I'd assume its not there because its not a limit they impose directly its something that number of r

Similar Threads

Similar Threads

Similar Threads