Hey folks, the content I'm storing on R2 is highly compressible. Like a 500kb vs 15mb. Will I want t

Hey folks, the content I'm storing on R2 is highly compressible. Like a 500kb vs 15mb.
Will I want to gzip this myself or is there a way to let R2 take care of it for me? And if I don't store the data gzipped, will I be charged?

VVenkat Dinavahi Hey folks, the content I'm storing on R2 is highly compressible. Like a 500kb vs...

dave•4/3/24, 2:55 PM

you gotta upload it gzip'd yourself afaik, pretty easy to do with a Worker (that's what we do).

dave•4/3/24, 2:56 PM

make sure to set

Content-Encoding: gzip

Content-Encoding: gzip

when uploading, that way CF can automatically decompress it for clients that don't support gzip too.

Venkat DinavahiOP•4/3/24, 2:57 PM

Ah interesting... so it would get stored gzipped and downloading it this will happen transparently?

Venkat DinavahiOP•4/3/24, 2:57 PM

Same if I fetch it using the R2 binding on the backend?

Venkat DinavahiOP•4/3/24, 2:57 PM

If so, very cool! I'm surprised I survived so long without understanding all these http details

VVenkat Dinavahi Ah interesting... so it would get stored gzipped and downloading it this will ha...

dave•4/3/24, 3:01 PM

yep, unless you set

cache-control: no-transform

cache-control: no-transform

, which blocks CF from decompressing it

prabha•4/3/24, 3:02 PM

Hi Community, How to solve this issue for presigned URL

Ddave yep, unless you set `cache-control: no-transform`, which blocks CF from decompre...

dave•4/3/24, 3:04 PM

@Venkat Dinavahi e.g. you can see this fails to decompress automatically, because I have

no-transform

no-transform

set. https://azure-di.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/2024-02-29-preview/prebuilt-read/a45b747b5ed4c37de569213aef10649a.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=022cbef64017d6c3154435aeb62aeac1%2F20240403%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20240403T150153Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b09dd0376defa5ce6ea683f2469304e877bc99d5626bb5c0266b3f3fa8d4a0d1

In some situations, like archival, you don't want files to be modified automatically (because their hash changes).

dave•4/3/24, 3:05 PM

I actually see a R2 "bug" now lol. The

Vary: Accept-Encoding

Vary: Accept-Encoding

in the response header is wrong afaik, since with

Cache-Control: no-transform

Cache-Control: no-transform

, the response won't vary at all.

Venkat DinavahiOP•4/3/24, 3:05 PM

Could you clarify what you mean by "modified automatically"?

Venkat DinavahiOP•4/3/24, 3:06 PM

Like cloudflare will transform (in this case un-gzip) the file?

VVenkat Dinavahi Like cloudflare will transform (in this case un-gzip) the file?

dave•4/3/24, 3:06 PM

yes, just not for the object URL I gave as an example

Original message was deleted

dave•4/3/24, 3:07 PM

but again it can be disabled with

Cache-Control: no-transform

Cache-Control: no-transform

Venkat DinavahiOP•4/3/24, 3:07 PM

amazing - then I'm not understanding why we would want to set no-transform

Venkat DinavahiOP•4/3/24, 3:07 PM

when is a hash necessary?

VVenkat Dinavahi Same if I fetch it using the R2 binding on the backend?

dave•4/3/24, 3:09 PM

if you want to set (or "change")

cache-control

cache-control

after you've already uploaded a bunch of files, you can use aws4fetch to make a presigned URL with

?response-content-encoding=blahblah

?response-content-encoding=blahblah

set to override it.

Ddave if you want to set (or "change") `cache-control` _after_ you've already uploaded...

dave•4/3/24, 3:11 PM

Here's an example signed URL that does an override:

https://6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/azure-di-source-files-enam/83d6300e2bb85d71b49bedd61f19fc37?response-cache-control=private%2C%20immutable%2C%20no-transform%2C%20max-age%3D31536000%2C%20stale-while-revalidate%3D31536000%2C%20stale-if-error%3D31536000%2C%20s-maxage%3D0&response-expires=Thu%2C%2001%20Jan%201970%2000%3A00%3A00%20GMT&X-Amz-Expires=604800&X-Amz-Date=20240401T000000Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b55f117ae28b9ec08c4b242ae09b712c%2F20240401%2Fauto%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5353652a3bb5083b4100c81c477726de0955a2f900c0fe4d9b406301dcfff896

*•4/3/24, 3:12 PM

that looks like a very short and simple URL

VVenkat Dinavahi when is a hash necessary?

dave•4/3/24, 3:12 PM

if you're storing the compressed data in other places, sometimes it can just be easier to not have automatic decompression so you can verify nothing is corrupted.

If you aren't storing stuff in multiple places with different S3 backends, you probably don't care.

Venkat DinavahiOP•4/3/24, 3:13 PM

ah got it, so you can fetch the thing that's actually stored on R2 and check the contents/hash of it

Venkat DinavahiOP•4/3/24, 3:14 PM

what would that mean for the checksum? if I want to send a sha1 checksum, is that going to be of the gzipped file or the ungzipped file?

Venkat DinavahiOP•4/3/24, 3:15 PM

Also, funny thing is I'm also dealing with storing OCR data

Pprabha Hi Community, How to solve this issue for presigned URL

prabha•4/3/24, 3:17 PM

Hi any one regarding this

VVenkat Dinavahi what would that mean for the checksum? if I want to send a sha1 checksum, is tha...

dave•4/3/24, 3:21 PM

it'd be the checksum of the gzipped file. But when your client downloads it, it won't have the same checksum as it'll be decompressed.

VVenkat Dinavahi Also, funny thing is I'm also dealing with storing OCR data 🙂

dave•4/3/24, 3:24 PM

Azure is the best I've seen so far, though their API has a lot of pitfails.. that's why we wrote a bunch of Workers stuff to make it sane. We've OCR'd a couple hundred thousand pages iirc with our setup.

Venkat DinavahiOP•4/3/24, 3:25 PM

I fell in love with the cloudflare platform.. having everything centralized DNS, workers, storage, etc has been a huge benefit

Venkat DinavahiOP•4/3/24, 3:26 PM

less infrastructure details to think about

**that looks like a very short and simple URL

dave•4/3/24, 3:26 PM

eh, that's presigned S3 URLs for you. The "shorter" version we made with public R2 buckets, a WAF rule, transform rules, and a redirect rule is https://azure-di-source-files.auto.ai.moda/83d6300e2bb85d71b49bedd61f19fc37?cache-control=private%2C+immutable%2C+no-transform%2C+max-age=31536000%2C+stale-while-revalidate=31536000%2C+stale-if-error=31536000%2C+s-maxage=0&expires=Thu%2C+01+Jan+1970+00:00:00+GMT&verify=1712102400-4wZJur5FPt%2FlgOxQfSXl4Yryh6J%2Bjsj0N482RnL7jNc=

Ddave eh, that's presigned S3 URLs for you. The "shorter" version we made with public ...

Baby Beel•4/3/24, 3:31 PM

if the verify changes, shouldn't this invalidate the cache?

BBaby Beel if the verify changes, shouldn't this invalidate the cache?

dave•4/3/24, 3:32 PM

it will only invalidate locally on your machine. It won't invalidate the cached R2 object in CF.

dave•4/3/24, 3:32 PM

and to reduce the amount of that happening locally, we also round the timestamps to the current day.

Ddave eh, that's presigned S3 URLs for you. The "shorter" version we made with public ...

*•4/3/24, 3:33 PM

i see

*•4/3/24, 3:33 PM

i was half joking cause idk what these URL really mean lol

**i was half joking cause idk what these URL really mean lol

dave•4/3/24, 3:33 PM

it's a fair question.

Venkat DinavahiOP•4/3/24, 3:36 PM

Anyone using checksums with R2? It's reliable?

VVenkat Dinavahi Anyone using checksums with R2? It's reliable?

dave•4/3/24, 3:36 PM

yeah it's reliable.

Venkat DinavahiOP•4/3/24, 3:36 PM

ok great

dave•4/3/24, 3:37 PM

i'll walk you through an example of gzip vs auto decompression and why it can matter for hashes

Ddave it's a fair question.

*•4/3/24, 3:37 PM

i just use cloudflare to hide my webserver IP and to speed things up

VVenkat Dinavahi what would that mean for the checksum? if I want to send a sha1 checksum, is tha...

dave•4/3/24, 3:39 PM

^ with no-transform, this is the output:

You can see the hash of the downloaded file matches the ETag perfectly.

dave•4/3/24, 3:40 PM

but, with automatic decompression, the downloaded file would have a hash of .

Venkat DinavahiOP•4/3/24, 3:41 PM

and if I'm understanding right, the etag means the client can skip requesting the body of the response if the etag matches a previously cached response?

VVenkat Dinavahi and if I'm understanding right, the etag means the client can skip requesting th...

dave•4/3/24, 3:42 PM

yes. If you're opening files in the browser, it'll handle the etag matching stuff for you.

Venkat DinavahiOP•4/3/24, 3:45 PM

I think I'm still confused haha. So no-transform is saying "don't ungzip this" for this case which means the md5 is calculated on the bytes that are actually stored.

Venkat DinavahiOP•4/3/24, 3:45 PM

Like what is the pipeline with transform vs no-transform

Venkat DinavahiOP•4/3/24, 3:46 PM

r2 => cloudflare => browser
somewhere in there the file is getting unzipped, but that would happen in the browser right?

Venkat DinavahiOP•4/3/24, 3:48 PM

on disk (r2) the file is stored gzipped the header is stored on r2 also
when browser requests it, the compressed file and the headers are proxied through cloudflare and sent to the browser, browser decompresses it because it has those headers

is my understanding correct?

VVenkat Dinavahi I think I'm still confused haha. So no-transform is saying "don't ungzip this" f...

dave•4/3/24, 3:49 PM

no-transform is telling cf "don't ungzip this"

Venkat DinavahiOP•4/3/24, 3:49 PM

ahhhh, so cloudflare ungzips it?

Venkat DinavahiOP•4/3/24, 3:50 PM

wouldn't that also have the downside that more data is transferred to the client? like the browser downloads 15mb instead of 500kb?