how would that work?

K

kian•10/26/23, 10:25 PM

you could tell rclone not to check if it already exists, or use R2’s migrator if they’re already on S3

Aalexandernst how would that work?

O

omar4289•10/26/23, 10:32 PM

If you look at r2 documentation, there is a flag in the r2 binding called onlyIf which takes some conditions as an argument. One of the conditions is unmodifiedSince (or updatedSince) which can be used to only import the file when it is not already in the bucket

A

alexandernstOP•10/26/23, 10:49 PM

oh, but then I’d need to “download” the file with a worker and store it in r2

A

alexandernstOP•10/26/23, 10:49 PM

basically the same thing as uploading it from somewhere else, just that the “work” is being inverted

J

JustinNoel•10/27/23, 12:10 AM

I have a public R2 bucket associated with my custom domain. It holds profile photos for my users. I use image resizing to get their profile photo from the r2 bucket and re-size it with query parameters.

I have a DNS CNAME for

profile-photos.my-domain.com

profile-photos.my-domain.com

profile-photos.my-domain.com

profile-photos.my-domain.com that points to the R2 bucket.

On my website, when a profile photo is to be displayed, the URL for the image looks like this:

URL 1:

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

https://my-domain.com/resources/profile-photos/FNS-2304

Then, I have a worker that basically proxies the URL above to my

profile-photos

profile-photos

profile-photos

profile-photos subdomain with a fetch request:

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

// The slug in this URL example is `FNS-2304`
const slug = c.req.param("photos-slug");
const { refresh } = c.req.query();
const refreshString = refresh ? `?refresh=${refresh}` : "";
const photoUrl = `https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/${slug}${refreshString}`;
return fetch(photoUrl, {})

So per my code above, URL 1 would query my R2 bucket like this:

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304

When the user changes their profile photo, I immediately request the same URL with a unique query parameter at the end to force the R2 bucket to give me the newest version that the user just uploaded:

URL 2:

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://my-domain.com/resources/profile-photos/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

So per my code above, URL 2 would query my R2 bucket like this:

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

https://profile-photos.my-domain.com/cdn-cgi/image/width=256,quality=99,format=auto/FNS-2304?refresh=01HDQ5X4V6G1FKFNOT89WAR1SC

When the user that uploaded their new profile photo reviews their profile, they see the new photo.

However, when other people view my users profile page, they still are using URL 1 so they're not seeing the newest version

. It takes some unknown amount of time open (several minutes) for the new photo to become the cached version.

How can I force R2 or Cloudflare cache to immediately eliminate any old versions when a new photo is uploaded?

B

Bruce Liu•10/27/23, 3:29 AM

I use JS to upload files to R2 on the front end. In order not to expose my KEY, I use pre-signed URLS, but I have a question, can pre-signed URLs upload large files? Because it does not resume uploading.

BBruce Liu I use JS to upload files to R2 on the front end. In order not to expose my KEY, ...

O

omar4289•10/27/23, 3:57 AM

I personally would not do that just with presigned URL because technically the user can upload hundreds of MB of files to your server through the same presigned URL. Currently, the max file size with presigned is 100MB for pro/free, 200 for business and 500 for entreprise

Aalexandernst oh, but then I’d need to “download” the file with a worker and store it in r2

O

omar4289•10/27/23, 3:59 AM

No, you don't need to download from a worker, you will just upload your files with the header "If-Unmodified-Since": "Tue, 28 Sep 2021 16:00:00 GMT"

JJustinNoel I have a public R2 bucket associated with my custom domain. It holds profile pho...

O

omar4289•10/27/23, 4:08 AM

Instead of dealing with user images as a one to one mapping with r2, I would put the new image under a different r2 path and update my user's record in db to point to the new r2 path - Would also be good to delete the old user's image on r2

I

iandk•10/27/23, 9:28 AM

Hi! Do I have to adjust the endpoint url in rclone when using the EU jurisdiction?
I'm having issues accessing a EU bucket.

A regular one works just fine

I

iandk•10/27/23, 9:29 AM

Okay nevermind, solved it myself.
eu.r2.cloudflarestorage.com does the job

I

iandk•10/27/23, 9:29 AM

However I'm seeing lots of internal errors when browsing the UI

JJustinNoel I have a public R2 bucket associated with my custom domain. It holds profile pho...

S

Sid•10/27/23, 2:33 PM

The issue here, IIUC is the fact that the CF cache is colo-local only (every CF data center has its own independent cache). Uploading a new picture and forcing cache to update does what you expect in the colo nearest to the uploader, but this change does not automatically propagate to other colos around the world. Omar’s solution works too, but since you’re already using Workers, check this out to see if it helps? https://developers.cloudflare.com/workers/learning/how-the-cache-works/#single-file-purge--assets-cached-by-a-worker

This also explains single-file purge: https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-single-file/

Purge by single-file · Cloudflare Cache (CDN) docs

With purge by single-file, cached resources are immediately removed from the stored assets in your Content Delivery Network (CDN) across all data …

How the Cache works · Cloudflare Workers docs

How Workers interacts with the Cloudflare cache.

Iiandk However I'm seeing lots of internal errors when browsing the UI

S

Sid•10/27/23, 2:33 PM

What page do you see these? Are they only showing up for jurisdictional buckets? I’m unable to repro this FWIW

SSid The issue here, IIUC is the fact that the CF cache is colo-local only (every CF ...

H

Hello, I’m Allie!•10/27/23, 3:45 PM

So wait, just to make sure, is a Colo a group of PoPs? Or a PoP a group of colos?

HHello, I’m Allie!So wait, just to make sure, is a Colo a group of PoPs? Or a PoP a group of colos...

S

Sid•10/27/23, 3:48 PM

A colo is a group of machines. So a colo / PoP / datacenter are all the same thing in this context.

SSid A colo is a group of machines. So a colo / PoP / datacenter are all the same thi...

H

Hello, I’m Allie!•10/27/23, 3:49 PM

So what is IAD then? A colo? A PoP?

K

kian•10/27/23, 3:49 PM

In the case of MCPs, cache is still per-colo so

lhr01

lhr01

and

lhr08

lhr08

would have different caches despite being apart of the same

LHR

LHR

PoP.

H

Hello, I’m Allie!•10/27/23, 3:50 PM

Oh, that's what they are called, MCPs...

K

kian•10/27/23, 3:50 PM

Some PoPs are MCPs, but not all PoPs with multiple colos are MCPs. :^)

HHello, I’m Allie!So what is IAD then? A colo? A PoP?

S

Sid•10/27/23, 3:51 PM

IAD is the name of Ashburn’s Colo if I understood your question correctly

Kkian Some PoPs are MCPs, but not all PoPs with multiple colos are MCPs. :^)

H

Hello, I’m Allie!•10/27/23, 3:51 PM

:myotlyikethis*

SSid IAD is the name of Ashburn’s Colo if I understood your question correctly

H

Hello, I’m Allie!•10/27/23, 3:51 PM

I mean, if

iad01

iad01

,

iad02

iad02

and

iad03

iad03

exist, what is the term for IAD as a whole?

H

Hello, I’m Allie!•10/27/23, 3:52 PM

Or any other PoP(?) with multiple Colos(?)

S

Sid•10/27/23, 3:52 PM

Colo = colocated machines. A set of servers in a building. This building is also a PoP (a point of presence)

Some places have multiple colos close to each other (like London). In that case, London is called an MCP (a Multiple Colo PoP)

HHello, I’m Allie!Or any other PoP(?) with multiple Colos(?)

S

Sid•10/27/23, 3:53 PM

There are quite a few, I don’t have an exhaustive list at the moment though.

SSid There are quite a few, I don’t have an exhaustive list at the moment though.

H

Hello, I’m Allie!•10/27/23, 3:53 PM

You answered it above with that response, thanks! Though, in what case would a PoP with multiple Colos not be an MCP?

S

Sid•10/27/23, 3:54 PM

And if it wasn’t obvious, a colo is named after the nearest airport IATA code

HHello, I’m Allie!You answered it above with that response, thanks! Though, in what case would a P...

S

Sid•10/27/23, 3:55 PM

I don’t think that would ever be the case (this is all nomenclature BTW, nothing to do with functionality)

S

Sid•10/27/23, 3:55 PM

Why do you ask?

SSid Why do you ask?

H

Hello, I’m Allie!•10/27/23, 3:55 PM

Just in response to this: https://canary.discord.com/channels/595317990191398933/940663374377783388/1167490608949104730

H

Hello, I’m Allie!•10/27/23, 3:56 PM

Also because generally when you get a colo in code, it only has the IATA, not the digits, so I was wondering if there was a better name for it

HHello, I’m Allie!Just in response to this: https://canary.discord.com/channels/595317990191398933...

S

Sid•10/27/23, 3:56 PM

Oh haha is that so @kian

K

kian•10/27/23, 3:59 PM

I was referring to the MCPs referenced in https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/#background, where they operate together as opposed to individual colos/PoPs

C

Colin•10/28/23, 12:07 AM

hi

we're seeing a decent amount of

put: We encountered an internal error. Please try again. (10001)

put: We encountered an internal error. Please try again. (10001)

when calling

R2Bucket.put

R2Bucket.put

C

Colin•10/28/23, 12:08 AM

We're using R2 to store file uploads from users (via a Worker). We're using the request body as a stream, so we're unable to retry (as the stream is already consumed + too big to cache in memory)

C

Colin•10/28/23, 12:08 AM

I understand intermittent failure is expected in a system like R2, but wondering if anyone has ideas for how to reduce these error rates, or if anyone from CF can take a look

U

Unsmart•10/28/23, 12:27 AM

What is the error rate you get with that? @Colin

From my own testing it seems the error rate it seems quite low

C

Colin•10/28/23, 12:27 AM

oh yeah it's definitely quite low, I see it at ~1%

C

Colin•10/28/23, 12:28 AM

but shows up in logs often enough I'd figure it's worth seeing what's going on

U

Unsmart•10/28/23, 12:28 AM

1% seems pretty high, whats your upload volume?

UUnsmart 1% seems pretty high, whats your upload volume?

C

Colin•10/28/23, 12:34 AM

I’d estimate we have ~80k object uploads per week

CColin I’d estimate we have ~80k object uploads per week

U

Unsmart•10/28/23, 12:39 AM

Hmm R2 should be able to handle 80k uploads per week just fine I do ~40k per day with only a 0.08% failure rate. Do you happen to know the average size of the files and do you know if errors happen more often in certain regions?

C

Colin•10/28/23, 12:41 AM

Yeah it's definitely not a crazy high volume, so I'm not super sure why

C

Colin•10/28/23, 12:41 AM

Average size of the file is <=50kb; unfortunately don't have a per-region breakdown

C

Colin•10/28/23, 12:42 AM

I'm wondering if it's users' connections timing out mid-stream or something

U

Unsmart•10/28/23, 12:50 AM

That could be possible not sure. 50kb is pretty small and should upload really quick so would be surprising if that many had their internet cut out during that

CColin I understand intermittent failure is expected in a system like R2, but wondering...

S

Sid•10/28/23, 1:50 AM

If they’ve started happening over the last couple of days, they may be related to the incident we had. Can you send over your account ID and possibly your bucket’s name?

UUnsmart That could be possible not sure. 50kb is pretty small and should upload really q...

C

Colin•10/28/23, 1:51 AM

Me too, just don't have any other ideas as to what it could be

SSid If they’ve started happening over the last couple of days, they may be related t...

C

Colin•10/28/23, 1:51 AM

Unfortunately this is a newer workload (~1.5 weeks) so don't have a huge baseline here. Is it ok if I add you to DM the bucket name?

Similar Threads

Similar Threads

Similar Threads