Looks like its the known issue with files that are too large to fit into cache, the workaround is to

EErisa Looks like its the known issue with files that are too large to fit into cache, ...

B

Bruce Liu•10/26/23, 9:48 AM

OK, the video file is 5GB in size. However, I'm going to try writing rules now, bypass caching, and try.

B

Bruce Liu•10/26/23, 9:59 AM

I bypassed all caches through PAGE RULES, but unfortunately, it didn't take effect~

B

Bruce Liu•10/26/23, 10:04 AM

The cache rules are set like this, I don’t know if it takes effect. But the problem still exists, and the request status code of 200 still appears.

BBruce Liu The cache rules are set like this, I don’t know if it takes effect. But the prob...

K

Karew•10/26/23, 10:22 AM

Did you clear your existing cache? Also you can try the new Trace tool to check if rules are being applied to a request https://blog.cloudflare.com/traffic-transparency-unleashing-cloudflare-trace/

B

Bruce Liu•10/26/23, 10:47 AM

https://kangeshu.xyz/loki.mp4 This is my video link. When I slide the progress bar, I can still see the 200 request status code. I am going crazy.

BBruce Liu The cache rules are set like this, I don’t know if it takes effect. But the prob...

E

ErisaOP•10/26/23, 11:06 AM

Cache rules dont support wildcard, use "contains" as the operator and

as the value

H

Hello, I’m Allie!•10/26/23, 11:27 AM

Given that I upload a PNG to this Worker, is there any reason why it would return corrupted?

export default <ExportedHandler<Environment>>{
    async fetch(req, env) {
        if (req.method === "POST") {
            return Response.json(await env.R2.put("file", req.body));
        }
        return new Response((await env.R2.get("file"))?.body, {
            headers: {
                "content-type": "image/png"
            }
        });
    }
}

export default <ExportedHandler<Environment>>{
    async fetch(req, env) {
        if (req.method === "POST") {
            return Response.json(await env.R2.put("file", req.body));
        }
        return new Response((await env.R2.get("file"))?.body, {
            headers: {
                "content-type": "image/png"
            }
        });
    }
}

export default <ExportedHandler<Environment>>{
    async fetch(req, env) {
        if (req.method === "POST") {
            return Response.json(await env.R2.put("file", req.body));
        }
        return new Response((await env.R2.get("file"))?.body, {
            headers: {
                "content-type": "image/png"
            }
        });
    }
}

export default <ExportedHandler<Environment>>{
    async fetch(req, env) {
        if (req.method === "POST") {
            return Response.json(await env.R2.put("file", req.body));
        }
        return new Response((await env.R2.get("file"))?.body, {
            headers: {
                "content-type": "image/png"
            }
        });
    }
}

EErisa Cache rules dont support wildcard, use "contains" as the operator and `/` as the...

B

Bruce Liu•10/26/23, 11:36 AM

Thank you! I solved this problem.

Original message was deleted

H

Hello, I’m Allie!•10/26/23, 11:39 AM

The image returned doesn't have a matching SHA-Sum, for one:

3be0c615e0158c0ba0110ba40f2de4db36e24c32  original.png
ca96b61d7c28300963e091080819f46221f34d4a  downloaded.png

3be0c615e0158c0ba0110ba40f2de4db36e24c32  original.png
ca96b61d7c28300963e091080819f46221f34d4a  downloaded.png

3be0c615e0158c0ba0110ba40f2de4db36e24c32  original.png
ca96b61d7c28300963e091080819f46221f34d4a  downloaded.png

3be0c615e0158c0ba0110ba40f2de4db36e24c32  original.png
ca96b61d7c28300963e091080819f46221f34d4a  downloaded.png

The other is that it can't be opened at all

H

Hello, I’m Allie!•10/26/23, 11:46 AM

Yeah, that was it...

H

Hello, I’m Allie!•10/26/23, 11:46 AM

Thanks!

B

Bruce Liu•10/26/23, 11:47 AM

My front-end uses S3 SDK JS to upload files to R2. But now I need to use a pre-signed URL to upload files. Am I unable to use the S3 client? Can I only use XHR or FETCH directly?

B

Bruce Liu•10/26/23, 11:49 AM

Because in the S3 SDK, there are multiple uploads, disconnection reconnection and other mechanisms. These are all functions that I really need, because I upload some video files, which are relatively large. But if I use S3 to upload directly, I have to Expose my KEY on the browser side. My project uses JAVA+SPRINGBOOT.

D

dave.swift•10/26/23, 12:57 PM

I am trying to stream content to R2

put

put

put

put API. However, I keep getting "Provided readable stream must have a known length (request/response body or readable half of FixedLengthStream)" error. I have detailed a basic code example at https://gist.github.com/e332890143f40d1803a8d2aed089c517. Has anyone run into this and/or have a solution. Much appreciated!

K

kian•10/26/23, 1:00 PM

You would need to buffer the compressed stream in memory so that the size is known

D

dave.swift•10/26/23, 1:01 PM

Unfortunately, it's too large to fit in memory.

K

kian•10/26/23, 1:01 PM

That’s the only option for Workers - R2 requires the size to be known

D

dave.swift•10/26/23, 1:03 PM

That's disappointing. I thought that was the main purpose of using streams was to not load it all in memory at once. I appreciate your help though.

D

dave.swift•10/26/23, 2:02 PM

For what it's worth, I think I found a viable workaround (though not ideal obviously). I am breifly storing the compressed file in cache, then streaming it to R2. (example: https://gist.github.com/e332890143f40d1803a8d2aed089c517)

S

Sinan Kochan•10/26/23, 3:08 PM

Hello, i am using my R2 bucket for a short time, even though my bucket is public i cannot reach assets in it. i get "ERR_SSL_PROTOCOL_ERROR" What can i do for it?

Ddave.swift For what it's worth, I think I found a viable workaround (though not ideal obvio...

S

Sid•10/26/23, 4:28 PM

Have you also considered using multipart uploads for this? Thinking out loud, you could buffer a few chunks in the Worker’s memory and batch them in an UploadPart request, then complete the upload when the stream ends (or abort it if the stream errors)

S

stillmotion•10/26/23, 8:22 PM

Anyone experienced agreessive image caching on Chrome that causes CORS errors? I changed my allowed origins in R2 to

to avoid them, but still seeing it intermittently. Only solution I've found so far is to "Disable cache" in the dev tools network tab, and that only works some of the time.

V

vuliad•10/26/23, 8:40 PM

Hi! We have some problems with R2 right now. From some origins R2 just timeouts.

V

vuliad•10/26/23, 8:40 PM

V

vuliad•10/26/23, 8:41 PM

Does anyone having same problems?

A

alexandernst•10/26/23, 9:10 PM

I need to upload a huge amount of (small) files to R2 (we're talking about several million). And I need to keep uploading newly generated files, while skipping the ones that were already uploaded.

What would it be a good way to achieve that?

if I understand the docs correctly, rclone can check if each file exists (with an API call) and upload it (with another API call), but this is basically the same as just uploading it directly (the files are just a couple of kb so the round trip for an extra API call is more expensive than just uploading the file). And that wastes a lot of time and resources. Not ideal.

Maybe R2 supports uploading a zip and unzipping it once uploaded?

Aalexandernst I need to upload a huge amount of (small) files to R2 (we're talking about sever...

H

Hello, I’m Allie!•10/26/23, 9:24 PM

Those two API calls ensure that you aren’t uploading a file that already exists

H

Hello, I’m Allie!•10/26/23, 9:24 PM

So if you want to pay Class A Ops for every file, you can probably make it do that

A

alexandernst•10/26/23, 9:25 PM

so this is basically a tradeoff between speed (less speed because of 1 extra API call), and cost (it will be cheaper because of Class A ops)

A

alexandernst•10/26/23, 9:26 PM

what about uploading a zip file and unzipping it in r2? Is that possible?

K

kian•10/26/23, 9:31 PM

Nope

Aalexandernst I need to upload a huge amount of (small) files to R2 (we're talking about sever...

O

omar4289•10/26/23, 10:14 PM

Will you be doing this through a worker ?

A

alexandernst•10/26/23, 10:23 PM

how would that work?

K

kian•10/26/23, 10:25 PM

you could tell rclone not to check if it already exists, or use R2’s migrator if they’re already on S3

Aalexandernst how would that work?

O

omar4289•10/26/23, 10:32 PM

If you look at r2 documentation, there is a flag in the r2 binding called onlyIf which takes some conditions as an argument. One of the conditions is unmodifiedSince (or updatedSince) which can be used to only import the file when it is not already in the bucket

A

alexandernst•10/26/23, 10:49 PM

oh, but then I’d need to “download” the file with a worker and store it in r2

A

alexandernst•10/26/23, 10:49 PM

basically the same thing as uploading it from somewhere else, just that the “work” is being inverted

J

JustinNoel•10/27/23, 12:10 AM

I have a public R2 bucket associated with my custom domain. It holds profile photos for my users. I use image resizing to get their profile photo from the r2 bucket and re-size it with query parameters.

I have a DNS CNAME for

profile-photos.my-domain.com

profile-photos.my-domain.com

profile-photos.my-domain.com

profile-photos.my-domain.com that points to the R2 bucket.

On my website, when a profile photo is to be displayed, the URL for the image looks like this:

URL 1:

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

Then, I have a worker that basically proxies the URL above to my

profile-photos

profile-photos

profile-photos

profile-photos subdomain with a fetch request:

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

So per my code above, URL 1 would query my R2 bucket like this:

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

When the user changes their profile photo, I immediately request the same URL with a unique query parameter at the end to force the R2 bucket to give me the newest version that the user just uploaded:

URL 2:

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

So per my code above, URL 2 would query my R2 bucket like this:

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

When the user that uploaded their new profile photo reviews their profile, they see the new photo.

However, when other people view my users profile page, they still are using URL 1 so they're not seeing the newest version

. It takes some unknown amount of time open (several minutes) for the new photo to become the cached version.

How can I force R2 or Cloudflare cache to immediately eliminate any old versions when a new photo is uploaded?

B

Bruce Liu•10/27/23, 3:29 AM

I use JS to upload files to R2 on the front end. In order not to expose my KEY, I use pre-signed URLS, but I have a question, can pre-signed URLs upload large files? Because it does not resume uploading.

BBruce Liu I use JS to upload files to R2 on the front end. In order not to expose my KEY, ...

O

omar4289•10/27/23, 3:57 AM

I personally would not do that just with presigned URL because technically the user can upload hundreds of MB of files to your server through the same presigned URL. Currently, the max file size with presigned is 100MB for pro/free, 200 for business and 500 for entreprise

Aalexandernst oh, but then I’d need to “download” the file with a worker and store it in r2

O

omar4289•10/27/23, 3:59 AM

No, you don't need to download from a worker, you will just upload your files with the header "If-Unmodified-Since": "Tue, 28 Sep 2021 16:00:00 GMT"

JJustinNoel I have a public R2 bucket associated with my custom domain. It holds profile pho...

O

omar4289•10/27/23, 4:08 AM

Instead of dealing with user images as a one to one mapping with r2, I would put the new image under a different r2 path and update my user's record in db to point to the new r2 path - Would also be good to delete the old user's image on r2

I

iandk•10/27/23, 9:28 AM

Hi! Do I have to adjust the endpoint url in rclone when using the EU jurisdiction?
I'm having issues accessing a EU bucket.

A regular one works just fine

I

iandk•10/27/23, 9:29 AM

Okay nevermind, solved it myself.
eu.r2.cloudflarestorage.com does the job

I

iandk•10/27/23, 9:29 AM

However I'm seeing lots of internal errors when browsing the UI

JJustinNoel I have a public R2 bucket associated with my custom domain. It holds profile pho...

S

Sid•10/27/23, 2:33 PM

The issue here, IIUC is the fact that the CF cache is colo-local only (every CF data center has its own independent cache). Uploading a new picture and forcing cache to update does what you expect in the colo nearest to the uploader, but this change does not automatically propagate to other colos around the world. Omar’s solution works too, but since you’re already using Workers, check this out to see if it helps? https://developers.cloudflare.com/workers/learning/how-the-cache-works/#single-file-purge--assets-cached-by-a-worker

This also explains single-file purge: https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-single-file/

Purge by single-file · Cloudflare Cache (CDN) docs

With purge by single-file, cached resources are immediately removed from the stored assets in your Content Delivery Network (CDN) across all data …

How the Cache works · Cloudflare Workers docs

How Workers interacts with the Cloudflare cache.

Iiandk However I'm seeing lots of internal errors when browsing the UI

S

Sid•10/27/23, 2:33 PM

What page do you see these? Are they only showing up for jurisdictional buckets? I’m unable to repro this FWIW

SSid The issue here, IIUC is the fact that the CF cache is colo-local only (every CF ...

H

Hello, I’m Allie!•10/27/23, 3:45 PM

So wait, just to make sure, is a Colo a group of PoPs? Or a PoP a group of colos?

HHello, I’m Allie!So wait, just to make sure, is a Colo a group of PoPs? Or a PoP a group of colos...

S

Sid•10/27/23, 3:48 PM

A colo is a group of machines. So a colo / PoP / datacenter are all the same thing in this context.

SSid A colo is a group of machines. So a colo / PoP / datacenter are all the same thi...

H

Hello, I’m Allie!•10/27/23, 3:49 PM

So what is IAD then? A colo? A PoP?

Looks like its the known issue with files that are too large to fit into cache, the workaround is to

Similar Threads

Similar Threads

Similar Threads