26 replies

Fetching compressed objects from origin

I've got a bunch of compressed objects in the origin (GCS bucket) and if I hit them directly (with curl using

--compressed

--compressed

--compressed

--compressed), they are returned to me gzipped.

Now, if I hit my worker with

--compressed

--compressed

--compressed

--compressed and it goes and fetches the same object from the origin, it comes back uncompressed. I remember reading somewhere that regardless of what headers are sent in the request to Cloudflare, it will always attempt to fetch from the origin using compression and then depending on the user request, it will return compressed or uncompressed, but that doesn't seem to be the case.

Any ideas? TIA.

gonzOP•6/5/23, 10:31 AM

BTW my current workaround is to send request.headers along when calling fetch() but I wonder why that's required.

kian•6/5/23, 12:04 PM

Sounds like standard Accept-EncodingAccept-Encoding

kian•6/5/23, 12:04 PM

If you don't indicate GZIP support with Accept-EncodingAccept-Encoding then it would be decompressed on-the-fly

gonzOP•6/5/23, 12:10 PM

@kiannh ok so sending request headers to the origin sounds reasonable?

kian•6/5/23, 12:12 PM

If you want the client's compression support to be respected then yeah

gonzOP•6/5/23, 12:13 PM

Thanks!

gonzOP•6/5/23, 12:14 PM

Some of the files are highly compressed (10x) so yeah we want to send them compressed if the client requests it

kian•6/5/23, 12:16 PM

The part you found about 'it will always attempt to fetch from the origin using compression and then depending on the user request, it will return compressed or uncompressed' holds true

kian•6/5/23, 12:16 PM

Your Worker is the 'user' in this scenario though

kian•6/5/23, 12:16 PM

You might think it is equivalent of Cloudflare talking to the origin but it's actually equivalent of a user talking to Cloudflare

gonzOP•6/5/23, 12:16 PM

Right yeah

gonzOP•6/5/23, 12:17 PM

So it probably came back to CF compressed but due to the lack of accept encoding, it got uncompressed on-the-fly

kian•6/5/23, 12:18 PM

https://kian.org.uk/cloudflares-transparent-decompression-how-does-it-work/ explains the behaviour of a client's Accept-EncodingAccept-Encoding but the tl;dr is Cloudflare requests compressed content and will decompress it on-the-fly back to the user if required

Ggonz So it probably came back to CF compressed but due to the lack of accept encoding...

kian•6/5/23, 12:18 PM

Yup

kian•6/5/23, 12:18 PM

It's so that even if you don't support compression as a client, you save on bandwidth costs at your origin as it would be requested compressed.

gonzOP•6/5/23, 12:20 PM

I guess CF doesn't compress on the fly then? Seeing as how CF requests the content without accept encoding so it got uncompressed, given that the original request had accept encoding gzip, it doesn't then compress it before sending to the end user?

kian•6/5/23, 12:27 PM

It will compress on-the-fly

kian•6/5/23, 12:27 PM

https://developers.cloudflare.com/support/speed/optimization-file-size/what-will-cloudflare-compress/

What will Cloudflare compress · Cloudflare Support docs

In addition to Cloudflare’s CDN caching static content and auto-minification of CSS, JS & HTML to speed up your site, Cloudflare also provides gzip …

gonzOP•6/5/23, 12:33 PM

@kiannh so hrm why isn't the content returned by fetch() getting compressed on the fly?

kian•6/5/23, 12:35 PM

kian•6/5/23, 12:35 PM

Sounds like an issue with your specific scenario/origin

gonzOP•6/5/23, 1:05 PM

the origin URL is:

https://storage.googleapis.com/about.gitlab-review.app/1006-qa-fix-color-on-links-for-campus-page/index.html

This is what fetch()fetch() is getting... if you hit that with curlcurl, it comes back gzipped. Not sure what else I'm doing wrong for this content not to get compressed on the fly... the content-type is set to text/htmltext/html so it should get compressed?

gonzOP•6/5/23, 1:13 PM

Cache-Control: public, max-age=60,no-transformCache-Control: public, max-age=60,no-transform

gonzOP•6/5/23, 1:13 PM

hrm the doc does say to set cache-control to no-transformno-transform to avoid compression

gonzOP•6/5/23, 1:17 PM

yep that did it.. removed no-transformno-transform and it gzipped fine

gonzOP•6/5/23, 1:38 PM

we use gsutilgsutil to recursively upload files to GCS and one of the arguments is -z css,html,js,txt-z css,html,js,txt, which gzips those file extensions before uploading. This apparently forcefully sets no-transformno-transform on those uploaded files.

Fetching compressed objects from origin

Similar Threads

Similar Threads

Similar Threads