Is there a way similar to S3 to limit the upload size for a multipart upload client-side in R2? In A

Is there a way similar to S3 to limit the upload size for a multipart upload client-side in R2? In AWS there seems to be a way to limit the size with an IAM role, which R2 doesn't have. Maybe I'm missing an option..

My current solution is to create the multipart upload and then generating the presigned URLs something like this:

// Size is sent over by the client. It is checked against the maximum allowed file size on the server
const chunkCount = Math.ceil(size / FILE_CHUNK_SIZE_BYTES);
const fileKey = "my/r2/file_key";
const uploadSessionId = "R2_MULTIPART_SESSION_ID";
const presignedUrls = [];

for(let x = 1; x <= chunkCount; x++) {
  presignedUrls.push(await r2.getPresignedChunkUrl(fileKey, uploadSessionId, x, (60 * 60 * 24), x < chunkCount ? FILE_CHUNK_SIZE_BYTES : (size % FILE_CHUNK_SIZE_BYTES)));
}

// Size is sent over by the client. It is checked against the maximum allowed file size on the server
const chunkCount = Math.ceil(size / FILE_CHUNK_SIZE_BYTES);
const fileKey = "my/r2/file_key";
const uploadSessionId = "R2_MULTIPART_SESSION_ID";
const presignedUrls = [];

for(let x = 1; x <= chunkCount; x++) {
  presignedUrls.push(await r2.getPresignedChunkUrl(fileKey, uploadSessionId, x, (60 * 60 * 24), x < chunkCount ? FILE_CHUNK_SIZE_BYTES : (size % FILE_CHUNK_SIZE_BYTES)));
}

// Size is sent over by the client. It is checked against the maximum allowed file size on the server
const chunkCount = Math.ceil(size / FILE_CHUNK_SIZE_BYTES);
const fileKey = "my/r2/file_key";
const uploadSessionId = "R2_MULTIPART_SESSION_ID";
const presignedUrls = [];

for(let x = 1; x <= chunkCount; x++) {
  presignedUrls.push(await r2.getPresignedChunkUrl(fileKey, uploadSessionId, x, (60 * 60 * 24), x < chunkCount ? FILE_CHUNK_SIZE_BYTES : (size % FILE_CHUNK_SIZE_BYTES)));
}

// Size is sent over by the client. It is checked against the maximum allowed file size on the server
const chunkCount = Math.ceil(size / FILE_CHUNK_SIZE_BYTES);
const fileKey = "my/r2/file_key";
const uploadSessionId = "R2_MULTIPART_SESSION_ID";
const presignedUrls = [];

for(let x = 1; x <= chunkCount; x++) {
  presignedUrls.push(await r2.getPresignedChunkUrl(fileKey, uploadSessionId, x, (60 * 60 * 24), x < chunkCount ? FILE_CHUNK_SIZE_BYTES : (size % FILE_CHUNK_SIZE_BYTES)));
}

async getPresignedChunkUrl(key: string, uploadSessionId: string, partNumber: number, expiresInSec: number, contentLength: number) {
    try {
      const command = new UploadPartCommand({
        Bucket: this.bucket,
        Key: key,
        UploadId: uploadSessionId,
        PartNumber: partNumber,
        ContentLength: contentLength
      });

      return await getSignedUrl(this.client, command, {expiresIn: expiresInSec});
    }
    catch (e: any) {
      console.error(e?.message);
    }
    return null;
  }

async getPresignedChunkUrl(key: string, uploadSessionId: string, partNumber: number, expiresInSec: number, contentLength: number) {
    try {
      const command = new UploadPartCommand({
        Bucket: this.bucket,
        Key: key,
        UploadId: uploadSessionId,
        PartNumber: partNumber,
        ContentLength: contentLength
      });

      return await getSignedUrl(this.client, command, {expiresIn: expiresInSec});
    }
    catch (e: any) {
      console.error(e?.message);
    }
    return null;
  }

async getPresignedChunkUrl(key: string, uploadSessionId: string, partNumber: number, expiresInSec: number, contentLength: number) {
    try {
      const command = new UploadPartCommand({
        Bucket: this.bucket,
        Key: key,
        UploadId: uploadSessionId,
        PartNumber: partNumber,
        ContentLength: contentLength
      });

      return await getSignedUrl(this.client, command, {expiresIn: expiresInSec});
    }
    catch (e: any) {
      console.error(e?.message);
    }
    return null;
  }

async getPresignedChunkUrl(key: string, uploadSessionId: string, partNumber: number, expiresInSec: number, contentLength: number) {
    try {
      const command = new UploadPartCommand({
        Bucket: this.bucket,
        Key: key,
        UploadId: uploadSessionId,
        PartNumber: partNumber,
        ContentLength: contentLength
      });

      return await getSignedUrl(this.client, command, {expiresIn: expiresInSec});
    }
    catch (e: any) {
      console.error(e?.message);
    }
    return null;
  }

Every chunk has the FILE_CHUNK_SIZE_BYTES as the ContentLength except the last one which is the remainder of bytes.

It works but it feels clumsy.

Evil Morty•11/3/23, 10:45 PM

Hello what is the problem here?

Pprender most of requests are 70-180ms which is also a bit slow for me, but sometimes it ...

Karew•11/4/23, 2:07 AM

Are the files really new or very cold? That is not unusual for the very first file reads

Karew•11/4/23, 2:07 AM

But also there is currently a company-wide outage incident, so maybe don't test extensively right now

Karew•11/4/23, 2:12 AM

Unfortuantely, latency is hard to diagnose. It can depend on a lot of factors at the time you make the request. Are you using a custom domain and the CDN? What kind of request are you making? Signed?

lottor•11/4/23, 2:45 AM

hi, r2 is down? im getting error 401 since 2 hours ago ... i cant view/upload files

matcha•11/4/23, 2:46 AM

Hey guys, are you having unauthorized too on cloudflare R2?

Mmatcha Hey guys, are you having unauthorized too on cloudflare R2?

lottor•11/4/23, 2:46 AM

my site has that error .. since 2 hours ago ..

Unsmart•11/4/23, 2:47 AM

R2 is very much still up for me are you using workers bindings or custom domains or s3 api?

matcha•11/4/23, 2:47 AM

I use s3 api, it's always return 401 unauthorized when submitting HEAD request

lottor•11/4/23, 2:48 AM

im using s3 api too

Unsmart•11/4/23, 2:48 AM

Hmm s3 api works fine for me. If you go to the cloudflare dash do you see the key you are using? Its possible the key you are using has expired

matcha•11/4/23, 2:49 AM

I had this issue raise to me since yesterday, the key has been rolled out multiple times but still having same response from r2

Unsmart•11/4/23, 2:51 AM

Due to the outage of cloudflares core datacenter its possible updates to api keys may have been delayed things are looking a lot better now so I would try generating a fresh key and seeing if that works. Otherwise you might have to wait until the incident is fully resolved as it could still be having issues syncing api key data to edge datacenters

matcha•11/4/23, 2:54 AM

Ah i see

lottor•11/4/23, 3:08 AM

i created a new key but it doesn't work.. I guess I'll have to wait..

grizzlebizzle•11/4/23, 4:23 AM

just started looking at using R2 for a project a week ago - is this kind of outage common? Looks like it started a couple days ago now and is still ongoing? I was able to sign up and create a bucket but getting all kinds of errors in the dashboard (internal error 10001) after doing that so I guess maybe I need to wait till the outage is resolved to get any further. I got a URL for the bucket but no key info or anything else - course I could be missing it since I haven't used it before. I'm assuming I can get that info in the area under where the URL is specified - which is blank for me.

fernando•11/4/23, 4:48 AM

Hi, I am trying to upload a file to a bucket using a presigned PutObjectCommand url, but I get this error

<Error>
    <Code>InternalError</Code>
    <Message>We encountered an internal error. Please try again.</Message>
</Error>

<Error>
    <Code>InternalError</Code>
    <Message>We encountered an internal error. Please try again.</Message>
</Error>

I am getting the URL like this:

getSignedUrl(s3, new PutObjectCommand({
        Bucket: bucket,
        Key: `${fileName}`
    }), {    expiresIn: 600 })

getSignedUrl(s3, new PutObjectCommand({
        Bucket: bucket,
        Key: `${fileName}`
    }), {    expiresIn: 600 })

and making an HTTP PUT Request on Postman. Is there anywhere where I can get more information about what I am doing wrong?

Kirmola•11/4/23, 5:00 AM

Looks like outage affected R2 too. All secret keys are returning unauthorised here I see

starina_cool•11/4/23, 5:27 AM

It is the biggest outage for the past 6 years as I remember.The outage itself is not a big deal. The problem is the amount of time it takes to fix it: https://www.cloudflarestatus.com/

Cloudflare Status

Welcome to Cloudflare's home for real-time and historical data on system performance.

Kirmola•11/4/23, 6:49 AM

UPDATE: R2 seems to be working now for uploading via s3.

Mmatcha Hey guys, are you having unauthorized too on cloudflare R2?

orkrist•11/4/23, 6:54 AM

yes, see my message above, it was terrible -- I destroyed half of my setup

thinking it was me.

orkrist•11/4/23, 6:55 AM

right bow it seems OK.

Sstarina_cool It is the biggest outage for the past 6 years as I remember.The outage itself is...

orkrist•11/4/23, 6:56 AM

the r2 status page showed fully operational status for all the time, it was highly misleading.

orkrist•11/4/23, 6:57 AM

I mean about 10h ago. the aws cli started working for me at that time I think, but I was unable to comment here (on fire a bit)

Oorkrist the r2 status page showed fully operational status for all the time, it was high...

Kirmola•11/4/23, 7:04 AM

neither api nor dashboard was working but statuspage showed it perfectly fine

Kirmola•11/4/23, 7:05 AM

I too was frustrated since yesterday,thinking i made some mistake, rolled keys like 100 times.

orkrist•11/4/23, 7:08 AM

hehe. same here @Aman Rawat ! man this cannot be like that, I actually do have production data there, and well, production systems stopped a bit.

Kirmola•11/4/23, 7:14 AM

mine was not in production but it still haunted me since yesterday

RRei My old token works, but the new token returns InternalError.

Kirmola•11/4/23, 7:14 AM

should be fine now..

PervySage•11/4/23, 8:18 AM

Is R2 still down? I just tried creating a new bucket and was bombarded with 10001 Errors on the dashboard. Also I'm a new user. Is it a reliable service or is it down often?

EW•11/4/23, 8:57 AM

I am using R2 with presigned URLs and they are not working since yesterday. However, Cloudflare says it's up and working. I could download my assets from the control panel but everything is private and using presigned URLs on the frontend. The server time is set correctly so that's not it. The site is https://www.epicwar.com/maps/

W7T2A•11/4/23, 9:15 AM

I'm also unable to auth against any operation on a newly created r2 bucket - always get unauthorized or server error

Gordon•11/4/23, 9:20 AM

I'm also getting multiple unauthorized errors, I've tried aws-sdk-js and aws-cli s3api

W7T2A•11/4/23, 9:21 AM

Yap same here

GGordon I'm also getting multiple unauthorized errors, I've tried aws-sdk-js and aws-cli...

Magic•11/4/23, 9:29 AM

same with s3fs, postman, rclone

cake•11/4/23, 10:25 AM

Our bots are failing our site upload, which we've never seen before. It started last night. We keep getting the following error:

aws.sdk.kotlin.services.s3.model.S3Exception: Unauthorizedaws.sdk.kotlin.services.s3.model.S3Exception: Unauthorized

Is R2 experiencing an outage?

cake•11/4/23, 10:26 AM

Were upload limits put in place recently?

EW•11/4/23, 10:29 AM

It's the same for all of us it seems.

EEW It's the same for all of us it seems.

kev-acOP•11/4/23, 10:31 AM

Yes it seems that way. I get Unauthorized errors on all buckets since (presumably) this night. Various backups to buckets are all missing this night. Rolling accesstokens does not help (in my case).

EW•11/4/23, 10:31 AM

At least I could download files from the admin panel to sort of verify that things weren't gone.

Is there a way similar to S3 to limit the upload size for a multipart upload client-side in R2? In A

Similar Threads