We are facing some issue while uploading file larger than 5Gb in the R2. We have migrated to R2 and

We are facing some issue while uploading file larger than 5Gb in the R2. We have migrated to R2 and we started facing issue, we cannot ask client to update their code to use multipart upload, also multipart upload is not that straight forward. We have already submitted the request from https://docs.google.com/forms/d/e/1FAIpQLSd_fwAVOboH9SlutMonzbhCxuuuOmiU1L_I5O2CFbXf_XXMRg/viewform?pli=1&pli=1 but no reply yet. We are facing too much issue due to this. We have use case of upload upto 15Gb. Could anybody from cloudflare team can help on this?

@Skye @Matt

Google Docs

Cloudflare Developer Platform: Limit Increase Request Form

Want to increase the limits of your current plan? Let us know what your requirements are here.

SShubham We are facing some issue while uploading file larger than 5Gb in the R2. We have...

ShubhamOP•1/22/24, 1:56 PM

Account ID: 83025b28472d6aa2bf5ae59f3724aa78

Erisa•1/22/24, 2:15 PM

Please don't ping people for this. If you need urgent help you will need to open a support ticket, Discord server assistance is community supported and best effort.
To my knowledge 5GB is a fixed limit that should not be too unusual for an S3-compatible service and S3 multipart uploads are effectively an industry standard when handling large uploads.

ShubhamOP•1/22/24, 2:49 PM

@Erisa | Support Engineer I raised the request using the request form and I tried opening support ticket but it is redirecting to discord for any R2 related issue. I understand that but we have migrated around 1TB of data to R2 and we are facing now issue which we were not facing with other provider. It was just a humble request to look into the request raised by Form as the business is getting impacted.

ShubhamOP•1/22/24, 2:55 PM

@Space - Ping in replies I am asking because there was option for that request, we were using already another provider where we didn't had issue and after migration it is very difficult to move back more that 1TB data. If it cannot be done then we will be stuck with multi part option only that I am aware.

Unsmart•1/22/24, 2:57 PM

Workers would have the same 5GB limit. Probably lower if they aren’t enterprise

Unsmart•1/22/24, 2:58 PM

Yes. The 5GB limit is the max you can proxy through cloudflare on ent

ShubhamOP•1/22/24, 2:58 PM

Exactly, before that we were streaming data with 7-8 GB

ShubhamOP•1/22/24, 2:59 PM

If this request can be done for increase we will be happy otherwise we need to involve many stakeholders. But Thank you anyways

Unsmart•1/22/24, 2:59 PM

5GB is the absolute max you have to use multipart

UUnsmart Yes. The 5GB limit is the max you can proxy through cloudflare on ent

Erisa•1/22/24, 3:22 PM

Proxied upload can be increased above 5GB on Ent (and the max without contacting support/success is 500MB), but you start running into tradeoffs and needing to disable body buffering when you get that high so its usually not worth it.

Scruffy•1/22/24, 5:36 PM

I've got a corrupted file upload issue. On the web UI the files are listed with their respective upload dates and sizes. When I open a specific object in the web UI it shows as having 0 Bytes in size and any attempt to access it via the public URL returns a 404

Scruffy•1/22/24, 5:39 PM

Scruffy•1/22/24, 5:40 PM

To be clear my bucket does have public access enabled and it works for all my other objects

SScruffy I've got a corrupted file upload issue. On the web UI the files are listed with ...

Matt Silverlock•1/22/24, 5:42 PM

Can you please open a Support ticket, including the account ID and bucket ID? That's the best place for us to investigate this — any suggestion of "corruption" is taken very seriously

MMatt Silverlock Can you please open a Support ticket, including the account ID and bucket ID? Th...

Scruffy•1/22/24, 5:42 PM

Sure i dont actually know how to open a support ticket

Unsmart•1/22/24, 5:42 PM

The issue is the hashtag... when accessing by url you need to percent encode %23%23

Unsmart•1/22/24, 5:43 PM

https://support.google.com/richmedia/answer/190941?hl=en

UUnsmart <https://support.google.com/richmedia/answer/190941?hl=en>

Scruffy•1/22/24, 5:43 PM

yep thats correct

Scruffy•1/22/24, 5:43 PM

Scruffy•1/22/24, 5:44 PM

would suggest fixing it on the cloudflare UI

Scruffy•1/22/24, 5:44 PM

Venkat Dinavahi•1/22/24, 6:26 PM

Hi has anyone succeeded in uploading to cloudflare R2 with a presigned url?

Venkat Dinavahi•1/22/24, 6:26 PM

If so, are there any working examples online for this? i've been struggling with a CORS issue. Seems to work for GET requests but PUT requests don't work.

Venkat Dinavahi•1/22/24, 6:28 PM

Or my ultimate goal is to be able to support uploads of up to 2GB but it appears that uploading to my cloudflare worker has request body limits.

SShubham We are facing some issue while uploading file larger than 5Gb in the R2. We have...

Venkat Dinavahi•1/22/24, 6:30 PM

How are you able to upload files even 2GB? I'm not able to figure out how to do this.

Hexstream•1/22/24, 7:47 PM

Hello, I'm getting unexpected errors while trying to retrieve stuff from R2, bizarrely cloudflarestatus does not seem to know about this at all...
Even downloading files directly from the R2 dashboard doesn't work.

Hexstream•1/22/24, 7:49 PM

get: The specified method is not allowed against this resource (10060)

Hexstream•1/22/24, 7:49 PM

Discord messages are too short for the full stacktrace, unfortunately.

Hexstream•1/22/24, 7:51 PM

Hum, I think redirects may be handled more strictly than before?

https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-10xxx-errors/#error-10060-missing-scheme-in-redirect-target-url

Hexstream•1/22/24, 7:53 PM

Stuff is normally accessed through a trivial worker that just retrieves the R2 stuff, but if I try to download stuff from the dashboard that doesn't work either.

Hexstream•1/22/24, 7:54 PM

I could post a worker URL, but that could be seen as a bit spammy I guess...

Unsmart•1/22/24, 7:54 PM

You can post the URL. Everything works fine for me so seems its most likely an issue on your end so code would be helpful also

Hexstream•1/22/24, 7:55 PM

https://ponies.hexstream.net/videos/brony-originals/misc/buttons-adventures-episode-pilot.mp4 would be an example URL.

Unsmart•1/22/24, 7:55 PM

and whats the code for how you are getting the object?

Hexstream•1/22/24, 7:55 PM

I haven't changed anything related to that recently, so this must be a new incompatibility of some sort.

Hexstream•1/22/24, 7:57 PM

Yeah I'll fetch the code, I haven't touched that one in a while...

Hexstream•1/22/24, 7:58 PM

function retrieve (url, env) {
    const path = url.hostname + url.pathname;
    const split = path.split("/");
    const tmpIndex = split.indexOf("tmp");
    if (tmpIndex >= 0) {
        split.splice(tmpIndex, 1);
        const bucket = env.TMP_BUCKET;
        const key = split.join("/");
        return bucket.get(key);
    } else {
        const bucket = env.BIG_BUCKET;
        const key = split.join("/");
        return bucket.get(key);
    }
}

export default {
  async fetch (request, env, context) {
      const cache = await caches.open("r2");
      const cachedResponse = await cache.match(request);
      if (cachedResponse)
          return cachedResponse;
      const object = await retrieve(new URL(request.url), env);
      if (object === null)
          return fetch(request);
      const body = await object.body;
      const headers = new Headers();
      object.writeHttpMetadata(headers);
      headers.set('etag', object.httpEtag);
      headers.append('Cache-Control', 's-maxage=691200');
      const response = new Response(body, { headers });
      const range = request.headers.get("Range");
      if (range === null) {
          context.waitUntil(cache.put(request, response.clone()));
          return response;
      } else {
          await cache.put(request, response);
          return cache.match(request);
      }
  }
};

function retrieve (url, env) {
    const path = url.hostname + url.pathname;
    const split = path.split("/");
    const tmpIndex = split.indexOf("tmp");
    if (tmpIndex >= 0) {
        split.splice(tmpIndex, 1);
        const bucket = env.TMP_BUCKET;
        const key = split.join("/");
        return bucket.get(key);
    } else {
        const bucket = env.BIG_BUCKET;
        const key = split.join("/");
        return bucket.get(key);
    }
}

export default {
  async fetch (request, env, context) {
      const cache = await caches.open("r2");
      const cachedResponse = await cache.match(request);
      if (cachedResponse)
          return cachedResponse;
      const object = await retrieve(new URL(request.url), env);
      if (object === null)
          return fetch(request);
      const body = await object.body;
      const headers = new Headers();
      object.writeHttpMetadata(headers);
      headers.set('etag', object.httpEtag);
      headers.append('Cache-Control', 's-maxage=691200');
      const response = new Response(body, { headers });
      const range = request.headers.get("Range");
      if (range === null) {
          context.waitUntil(cache.put(request, response.clone()));
          return response;
      } else {
          await cache.put(request, response);
          return cache.match(request);
      }
  }
};

Hexstream•1/22/24, 8:01 PM

But logically this is probably not related to a problem with that code, since it hasn't changed in ages and I can't download the file directly from Cloudflare's R2 dashboard either, unless that's an unrelated issue...

Unsmart•1/22/24, 8:02 PM

whats the error you get when trying to download from the dashboard?

Hexstream•1/22/24, 8:03 PM

It just says "Download failed. Please try again.".

Unsmart•1/22/24, 8:04 PM

Not sure I would open a support ticket

Hexstream•1/22/24, 8:05 PM

I'm on the free plan. The support wizard told me to post about this here.

Hexstream•1/22/24, 8:06 PM

Hexstream•1/22/24, 8:07 PM

Perhaps I could try posting in the community support forum thingy...

Hexstream•1/22/24, 8:07 PM

I would be pretty shocked if I'm the only one affected by this...

Unsmart•1/22/24, 8:09 PM

I'd find it very likely you are the only one otherwise I would expect a lot of reports here and on the community yet there are none and things work just fine for me on all of my buckets. Probably some configuration option set to a bad value but not sure

Hexstream•1/22/24, 8:10 PM

That's the weird thing, nothing changed, basically...
These errors just started popping up out of nowhere.

Unsmart•1/22/24, 8:10 PM

R2 employees do read this channel from time to time so they would best be able to assist when they see it

Hexstream•1/22/24, 8:12 PM

Yeah, normally I would use my tmp bucket but that won't work right now... I'll upload the full stacktrace somewhere.
But again, this also fails from the dashboard directly, so clearly unrelated to the worker trying to fetch.
(Unless the dashboard download error is unrelated...)

Hexstream•1/22/24, 8:13 PM

No, since yesterday at least.

We are facing some issue while uploading file larger than 5Gb in the R2. We have migrated to R2 and

Similar Threads

Similar Threads

Similar Threads