if you want to set (or "change") `cache-control` _after_ you've already uploaded a bunch of files, y

if you want to set (or "change")

cache-control

cache-control

after you've already uploaded a bunch of files, you can use aws4fetch to make a presigned URL with

?response-content-encoding=blahblah

?response-content-encoding=blahblah

set to override it.

Ddave if you want to set (or "change") `cache-control` _after_ you've already uploaded...

daveOP•4/3/24, 3:11 PM

Here's an example signed URL that does an override:

https://6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/azure-di-source-files-enam/83d6300e2bb85d71b49bedd61f19fc37?response-cache-control=private%2C%20immutable%2C%20no-transform%2C%20max-age%3D31536000%2C%20stale-while-revalidate%3D31536000%2C%20stale-if-error%3D31536000%2C%20s-maxage%3D0&response-expires=Thu%2C%2001%20Jan%201970%2000%3A00%3A00%20GMT&X-Amz-Expires=604800&X-Amz-Date=20240401T000000Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=b55f117ae28b9ec08c4b242ae09b712c%2F20240401%2Fauto%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5353652a3bb5083b4100c81c477726de0955a2f900c0fe4d9b406301dcfff896

*•4/3/24, 3:12 PM

that looks like a very short and simple URL

VVenkat Dinavahi when is a hash necessary?

daveOP•4/3/24, 3:12 PM

if you're storing the compressed data in other places, sometimes it can just be easier to not have automatic decompression so you can verify nothing is corrupted.

If you aren't storing stuff in multiple places with different S3 backends, you probably don't care.

Venkat Dinavahi•4/3/24, 3:13 PM

ah got it, so you can fetch the thing that's actually stored on R2 and check the contents/hash of it

Venkat Dinavahi•4/3/24, 3:14 PM

what would that mean for the checksum? if I want to send a sha1 checksum, is that going to be of the gzipped file or the ungzipped file?

Venkat Dinavahi•4/3/24, 3:15 PM

Also, funny thing is I'm also dealing with storing OCR data

Pprabha Hi Community, How to solve this issue for presigned URL

prabha•4/3/24, 3:17 PM

Hi any one regarding this

VVenkat Dinavahi what would that mean for the checksum? if I want to send a sha1 checksum, is tha...

daveOP•4/3/24, 3:21 PM

it'd be the checksum of the gzipped file. But when your client downloads it, it won't have the same checksum as it'll be decompressed.

VVenkat Dinavahi Also, funny thing is I'm also dealing with storing OCR data 🙂

daveOP•4/3/24, 3:24 PM

Azure is the best I've seen so far, though their API has a lot of pitfails.. that's why we wrote a bunch of Workers stuff to make it sane. We've OCR'd a couple hundred thousand pages iirc with our setup.

Venkat Dinavahi•4/3/24, 3:25 PM

I fell in love with the cloudflare platform.. having everything centralized DNS, workers, storage, etc has been a huge benefit

Venkat Dinavahi•4/3/24, 3:26 PM

less infrastructure details to think about

**that looks like a very short and simple URL

daveOP•4/3/24, 3:26 PM

eh, that's presigned S3 URLs for you. The "shorter" version we made with public R2 buckets, a WAF rule, transform rules, and a redirect rule is https://azure-di-source-files.auto.ai.moda/83d6300e2bb85d71b49bedd61f19fc37?cache-control=private%2C+immutable%2C+no-transform%2C+max-age=31536000%2C+stale-while-revalidate=31536000%2C+stale-if-error=31536000%2C+s-maxage=0&expires=Thu%2C+01+Jan+1970+00:00:00+GMT&verify=1712102400-4wZJur5FPt%2FlgOxQfSXl4Yryh6J%2Bjsj0N482RnL7jNc=

Ddave eh, that's presigned S3 URLs for you. The "shorter" version we made with public ...

Baby Beel•4/3/24, 3:31 PM

if the verify changes, shouldn't this invalidate the cache?

BBaby Beel if the verify changes, shouldn't this invalidate the cache?

daveOP•4/3/24, 3:32 PM

it will only invalidate locally on your machine. It won't invalidate the cached R2 object in CF.

daveOP•4/3/24, 3:32 PM

and to reduce the amount of that happening locally, we also round the timestamps to the current day.

Ddave eh, that's presigned S3 URLs for you. The "shorter" version we made with public ...

*•4/3/24, 3:33 PM

i see

*•4/3/24, 3:33 PM

i was half joking cause idk what these URL really mean lol

**i was half joking cause idk what these URL really mean lol

daveOP•4/3/24, 3:33 PM

it's a fair question.

Venkat Dinavahi•4/3/24, 3:36 PM

Anyone using checksums with R2? It's reliable?

VVenkat Dinavahi Anyone using checksums with R2? It's reliable?

daveOP•4/3/24, 3:36 PM

yeah it's reliable.

Venkat Dinavahi•4/3/24, 3:36 PM

ok great

daveOP•4/3/24, 3:37 PM

i'll walk you through an example of gzip vs auto decompression and why it can matter for hashes

Ddave it's a fair question.

*•4/3/24, 3:37 PM

i just use cloudflare to hide my webserver IP and to speed things up

VVenkat Dinavahi what would that mean for the checksum? if I want to send a sha1 checksum, is tha...

daveOP•4/3/24, 3:39 PM

curl "https://azure-di.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/2024-02-29-preview/prebuilt-read/a45b747b5ed4c37de569213aef10649a.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=022cbef64017d6c3154435aeb62aeac1%2F20240403%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20240403T153734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f048cb9c6827fd0af747e9f0dd9ea1b96fdee596db47a211391291ada5d723fe" -v -o - | md5sum

curl "https://azure-di.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/2024-02-29-preview/prebuilt-read/a45b747b5ed4c37de569213aef10649a.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=022cbef64017d6c3154435aeb62aeac1%2F20240403%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20240403T153734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=f048cb9c6827fd0af747e9f0dd9ea1b96fdee596db47a211391291ada5d723fe" -v -o - | md5sum

^ with no-transform, this is the output:

...
< Cache-Control: public, immutable, no-transform, max-age=31536000, s-maxage=31536000
< Content-Encoding: gzip
< ETag: "6be74e42ca7699d0b878d028eec8194f"
...
<
{ [786 bytes data]
100 15281  100 15281    0     0  63289      0 --:--:-- --:--:-- --:--:-- 63406
* Connection #0 to host azure-di.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com left intact
6be74e42ca7699d0b878d028eec8194f  -

...
< Cache-Control: public, immutable, no-transform, max-age=31536000, s-maxage=31536000
< Content-Encoding: gzip
< ETag: "6be74e42ca7699d0b878d028eec8194f"
...
<
{ [786 bytes data]
100 15281  100 15281    0     0  63289      0 --:--:-- --:--:-- --:--:-- 63406
* Connection #0 to host azure-di.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com left intact
6be74e42ca7699d0b878d028eec8194f  -

You can see the hash of the downloaded file matches the ETag perfectly.

daveOP•4/3/24, 3:40 PM

but, with automatic decompression, the downloaded file would have a hash of

a1f2837190f85fdffa030296859eb9b1

a1f2837190f85fdffa030296859eb9b1

Venkat Dinavahi•4/3/24, 3:41 PM

and if I'm understanding right, the etag means the client can skip requesting the body of the response if the etag matches a previously cached response?

VVenkat Dinavahi and if I'm understanding right, the etag means the client can skip requesting th...

daveOP•4/3/24, 3:42 PM

yes. If you're opening files in the browser, it'll handle the etag matching stuff for you.

Venkat Dinavahi•4/3/24, 3:45 PM

I think I'm still confused haha. So no-transform is saying "don't ungzip this" for this case which means the md5 is calculated on the bytes that are actually stored.

Venkat Dinavahi•4/3/24, 3:45 PM

Like what is the pipeline with transform vs no-transform

Venkat Dinavahi•4/3/24, 3:46 PM

r2 => cloudflare => browser
somewhere in there the file is getting unzipped, but that would happen in the browser right?

Venkat Dinavahi•4/3/24, 3:48 PM

on disk (r2) the file is stored gzipped the header

content-encoding: gzip

content-encoding: gzip

is stored on r2 also
when browser requests it, the compressed file and the headers are proxied through cloudflare and sent to the browser, browser decompresses it because it has those headers

is my understanding correct?

VVenkat Dinavahi I think I'm still confused haha. So no-transform is saying "don't ungzip this" f...

daveOP•4/3/24, 3:49 PM

no-transform is telling cf "don't ungzip this"

Venkat Dinavahi•4/3/24, 3:49 PM

ahhhh, so cloudflare ungzips it?

Venkat Dinavahi•4/3/24, 3:50 PM

wouldn't that also have the downside that more data is transferred to the client? like the browser downloads 15mb instead of 500kb?

VVenkat Dinavahi r2 => cloudflare => browser somewhere in there the file is getting unzipped, but...

daveOP•4/3/24, 3:50 PM

with no-transform:

r2 (gzip) => cloudflare => browser (ungzip)

without no-transform:

r2 (gzip) => cloudflare (ungzip and possibly recompress depending on the browser) => browser

VVenkat Dinavahi ahhhh, so cloudflare ungzips it?

daveOP•4/3/24, 3:50 PM

it'll recompress it to something like

br

br

if the browser supports it

daveOP•4/3/24, 3:50 PM

if the browser only supports gzip, then iirc it'll just pipe it through

Venkat Dinavahi•4/3/24, 3:51 PM

ohhh ok but it's up to CF what to do so it not good to rely on that

Venkat Dinavahi•4/3/24, 3:51 PM

no-transform says don't touch it just forward it to the client

Venkat Dinavahi•4/3/24, 3:54 PM

thanks for clarifying that

VVenkat Dinavahi ohhh ok but it's up to CF what to do so it not good to rely on that

daveOP•4/3/24, 3:59 PM

wellllll CF does do a great job imo. It's just really dependent on your use-case. Because OCR stuff is expensive (ish), I want to make sure I don't resubmit the same file that we've already processed because the hash changed.

Original message was deleted

Chaika•4/3/24, 4:01 PM

Data retrieval is charged per GB when data in the Infrequent Access storage class is retrieved and is what allows us to provide storage at a lower price. It reflects the additional computational resources required to fetch data from underlying storage optimized for less frequent access.

Chaika•4/3/24, 4:02 PM

afaik it's every time for a file in the IA tier

Chaika•4/3/24, 4:02 PM

why we should consider it different from egress pricing

don't think you should, other then for the fact egress pricing is normally talked about as a network concept and GCS for example charges both egress + retrieval for nearline. It is the cost you'll pay for retrieving a file.

VVenkat Dinavahi Also, funny thing is I'm also dealing with storing OCR data 🙂

daveOP•4/3/24, 4:03 PM

which OCR API are you using?

EErisa @R2 Object Storage - https://blog.cloudflare.com/r2-events-gcs-migration-infrequ...

Virtuous•4/3/24, 4:05 PM

wow perfect timing

adjlsncm•4/3/24, 5:41 PM

does anybody have any idea why im getting a type error when trying to upload an image to R2?

adjlsncm•4/3/24, 5:41 PM

kian•4/3/24, 6:25 PM

What is the error?