Caching by headers?

Hi, is it possible to cache pages based on the response headers or similar? We have some "static" pages on our site that we'd like to cache in Cloudflare, but we don't know the URLs (as we support custom CNAMEs) so I'm struggling to find a way to "Cache everything" without using Page Rules? I tried using a worker, but I couldn't seem to get that working either...
57 Replies
ryanbadger
ryanbadger7mo ago
forgot to mention, we set some headers on these pages hoping we could either encourage CF to cache them, or we could use them in workers to cache the response: // Set header to let Cloudflare know this request is static content and should be cached $CI->output->set_header('X-Static-Site: 1'); // Set Cache-Control header $CI->output->set_header('Cache-Control: public, max-age=60'); // Optionally set Expires header $expiresDate = gmdate('D, d M Y H:i:s', time() + 60) . ' GMT'; $CI->output->set_header("Expires: $expiresDate"); // Ensure there's no conflicting Pragma header $CI->output->set_header('Pragma: cache'); it's a static html page, do you mean I need to explicitly set the content type or something? here's my response headers im getting: Access-Control-Expose-Headers: X-Country, X-Language Alt-Svc: h3=":443"; ma=86400 Cache-Control: public, max-age=60 Cf-Cache-Status: DYNAMIC Cf-Ray: 8287ad8dee1e2df8-DPS Content-Encoding: br Content-Type: text/html; charset=UTF-8 Date: Sun, 19 Nov 2023 10:12:59 GMT Expires: Sun, 19 Nov 2023 10:13:59 GMT Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Pragma: cache Report-To: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=XH%2FPJXa8RpPl5J19gE1EXyoybmxIyuHMZ228uI15PGN0hGhw1Wa1VbDzShravyn2X7gUsiLtda3HvxFF%2FHxRCcQlifiQtMxtqV09wN5kDazRHBQmXo7NHnqyrhQXgZO8SuWNTA%3D%3D"}],"group":"cf-nel","max_age":604800} Server: cloudflare Set-Cookie: ci_sessions=xxx; expires=Mon, 20-Nov-2023 10:12:59 GMT; Max-Age=86400; path=/; secure; HttpOnly Strict-Transport-Security: max-age=15552000; includeSubDomains; preload Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Country: ID X-Language: en-GB X-Static-Site: 1
Hello, I’m Allie!
Do you have a Cache Everything rule set?
ryanbadger
ryanbadger7mo ago
in page rules? no the URLs are dynamic so i cant use page rules (hence me trying to do it via headers or workers)
ryanbadger
ryanbadger7mo ago
thanks but i dont think that will work for this page? https://demo.shoprocket.io/
Green Fingers Online Store
Browse & buy online from Green Fingers
ryanbadger
ryanbadger7mo ago
we also support custom cnames, so it could also be something liek https://store.mahakatha.com/
Mahakatha Meditation Online Store
Official Store of Mahakatha Meditation Mantras and Ancient Chants from India, Shiva Mantras, Ganesha Mantras and more
ryanbadger
ryanbadger7mo ago
i've been trying with workers all day and i THINK I have a solution its not perfect that's what i thought, but i cant figure out how in that link you sent, you say: "You can add a cache rule to force CF to cache stuff, it still wont cache dynamic content but will attempt to cache everything in caching/cache-rules/new" pretty sure i tried that but let me try again
ryanbadger
ryanbadger7mo ago
would you mind checking my headers here: https://demo.shoprocket.io/ are they correct?
Green Fingers Online Store
Browse & buy online from Green Fingers
ryanbadger
ryanbadger7mo ago
like this?
No description
ryanbadger
ryanbadger7mo ago
i used a != to apply to all requests because i wont know the hostnames (we support custom cnames) is it wrong?
ryanbadger
ryanbadger7mo ago
hows this?
No description
ryanbadger
ryanbadger7mo ago
whoops sorry, 1 moment
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
like this? no effect
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
ive been on/off trying to get this workign with CF for a couple of years now it never does every time i get close i find some small print somehwere that say it requires an enterprise plan i cant do that because then it will cache every response from our zone (if it caches at all) we have APIs etc that must not be cached i'll try...
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
how dot CF know whats dynamic? back to this again
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
welp it worked and cahced our APIs great 🥲 yes json e.g ** https://api.shoprocket.io/storefront/v1/stores/sr_live_pk_0ac696797e14d49ca01f523ffb8d962e670f/products?search=&sort=product_order&page=1&limit=12 i just disbaled it and had to "purge everything" because i disabled it but it was caching yeh and age
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
no you;re just seeing the old cache data i think basically, when i enable "Use cache-control header if present" - no cache with "Ignore cache-control header and use this TTL" - cache, but caches everything, even things i dong want cached
ryanbadger
ryanbadger7mo ago
i just changed it to this
No description
Hello, I’m Allie!
That's not what eligible for cache does Eligible for cache just disables the filetype check(or any other default checks)
ryanbadger
ryanbadger7mo ago
right so....its not doable?
Hello, I’m Allie!
What you have above should be fine
ryanbadger
ryanbadger7mo ago
doesnt seem to cache at all
ryanbadger
ryanbadger7mo ago
maybe i havent set my headers right? https://f90.shoprocket.io/
F90 Football Academy Store Online Store
Browse & buy online from F90 Football Academy Store
Hello, I’m Allie!
I'm getting HIT on the demo page
ryanbadger
ryanbadger7mo ago
Access-Control-Expose-Headers: X-Country, X-Language Alt-Svc: h3=":443"; ma=86400 Cache-Control: public, max-age=600 Cf-Cache-Status: BYPASS Cf-Ray: 828855f37c2d2df9-DPS Content-Encoding: br yeh the demo one is now cached with CF, i cant seem to purge it but no other urls are caching
ryanbadger
ryanbadger7mo ago
Dark Stry Online Store
Experience places differently with our self-guided walking and driving tours in Perth and the Coral Coast, Western Australia.
ryanbadger
ryanbadger7mo ago
Cache-Control: public, max-age=600 Cf-Cache-Status: BYPASS
Hello, I’m Allie!
What happens if you trace it? https://dash.cloudflare.com/?to=/:account/trace Does it show any Rules being applied?
ryanbadger
ryanbadger7mo ago
checking
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
do you think the transform rules are preventing it being cached?
ryanbadger
ryanbadger7mo ago
No description
Hello, I’m Allie!
It shouldn't If you click View Rule Detail on use origin, what does it show?
ryanbadger
ryanbadger7mo ago
{ "step_name": "2bb8e71132f1486baccc181d2d25bf30", "type": "rule", "matched": true, "action_parameter": { "cache": true, "edge_ttl": { "mode": "respect_origin" } }, "expression": "(http.host ne "foo")", "description": "use origin ", "action": "set_cache_settings" }
ryanbadger
ryanbadger7mo ago
i didabled those header rules just to rule them out
No description
ryanbadger
ryanbadger7mo ago
no change Alt-Svc: h3=":443"; ma=86400 Cache-Control: public, max-age=600 Cf-Cache-Status: BYPASS Cf-Ray: 828863155a342df9-DPS Content-Encoding: br Content-Type: text/html; charset=UTF-8 Date: Sun, 19 Nov 2023 12:16:54 GMT Expires: Sun, 19 Nov 2023 12:26:54 GMT Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Pragma: cache status is alwasy BYPASS or DYNAMIC
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
i also have smart tiered cache enabled and cache reserve
ryanbadger
ryanbadger7mo ago
i have that same config:
No description
ryanbadger
ryanbadger7mo ago
yup - as expected, thats just page rules which i cannot use you cant do it by resposne header only request header and i dont know what URLs need caching until the response is available origin yeh it doesnt i dont believe it ever has its been around 15 mins already no change
ryanbadger
ryanbadger7mo ago
No description
ryanbadger
ryanbadger7mo ago
No description
Hello, I’m Allie!
Think I realised the issue:
Cloudflare also sets BYPASS when your origin web server sends cookies in the response header.
ryanbadger
ryanbadger7mo ago
🥲 yeh i cant set in CF because i dont know the URLs
Hello, I’m Allie!
Can you get your origin to stop serving cookies?
ryanbadger
ryanbadger7mo ago
possibly - i'll need to dig into that but at least now i now why it never cached for me! thanks
Hello, I’m Allie!
Header Transforms run closer to the origin than Cache Rules, so if you can scrub the Cookie Header there, then it might still be able to cache Cloudflare Cookies shouldn't™️ be counted for cookies At least in terms of cachability
ryanbadger
ryanbadger7mo ago
done!
ryanbadger
ryanbadger7mo ago
Green Fingers Online Store
Browse & buy online from Green Fingers
ryanbadger
ryanbadger7mo ago
removed cookies from our hosted stores (they weren't need anyway) thanks everyone 🙏