Hiya, We are creating our caching system at the moment, I'm wondering, is there a way to tell cloud

Hiya,
We are creating our caching system at the moment, I'm wondering, is there a way to tell cloudflare; "go fetch all the urls in this path" in order for to cache it prior to any user visiting it?
If so, would it work for updates? I don't mind creating a microservice that does it, just wondering if anyone can lead me the right direction for that and if there's any API support for that

RRapid Hiya, We are creating our caching system at the moment, I'm wondering, is there...

Chaika•1/15/24, 5:45 PM

There isn't, and I would keep in mind too Cloudflare cache by default is heavily partitioned, there's one cache per colo/datacenter. That would be a lot of requests. Enterprise has something called Prefetch URLs (https://developers.cloudflare.com/speed/optimization/content/prefetch-urls/), which uses link response headers with prefetch that it'll bring into cache if responded with. (i.e user requests document html, it responds with link headers for assets, CF pre-fetches those before the user does)

You have Tiered Cache as well which for free you get the Smart Topology, which selects a CF location closest to your origin to check for cache, i.e User -> Local CF Datacenter/colo -> Smart Tiered Caching Selected -> Real Origin. If your goal is high cache % at the cost of some latency and you're not using an anycast service/something that lives in more then one location, Smart Tiered Caching can help.

(Edit: Enterprise has other topologies for tiered caching if you are ENT, that can work with even anycast origins in multiple locations and a middle tier option to further increase cache % with less latency)

Chaika•1/15/24, 5:47 PM

As for APIs there is a Purge API: https://developers.cloudflare.com/api/operations/zone-purge#purge-cached-content-by-url
You could have really long Cache TTL (keeping in mind it's a maximum and not guaranteed to be in cache for that long), and then call the purge API when updating content to try to increase cache % with still getting instant updates

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

RapidOP•1/15/24, 5:54 PM

So lets try it with an example. Our servers are based at NVirginia (nothing new :)).
1. We have around 4,000 static pages that barely (if ever) change.
2. We have our main page which gets exposure to many users and as there are more cache hits it increases cache time

I want to update my cache on my main page every 3mins for example (We are using the workers method to serve cached content with real-time fetched data). Which means every 3 mins I want to kind of "push" and update for the cached data.

With the static pages, lets say one of my writers makes an update, I want to purge it obviously, but then I want to create a new cache that sets a proper ageage for it, so incase google comes and tries to check it, it will be fetched from cache, and not access the server.

RapidOP•1/15/24, 5:55 PM

Hopefully this can provide more info on what I'm looking for, I would say google/client would be the same, since I want it to be as fast as possible obviously. Wouldn't it be possible to send a list of URLs to get cached? Like 4K URLs, and set a time (1 day) for example for caching. Does age even play a role here?

RRapid Hopefully this can provide more info on what I'm looking for, I would say google...

Chaika•1/15/24, 5:58 PM

Send where? There's no API you can send items to for Cloudflare to fetch & cache for you.
You could do a local fetch, but keeping in mind what I said above, without tiered cache on that would just populate the local cf datacenter cache.
There's no way to fill up all of the caches globally without having requests to all of those

Chaika•1/15/24, 5:59 PM

There is a min. edge cache ttl too. https://developers.cloudflare.com/cache/how-to/edge-browser-cache-ttl/
For Free, 2 hours, Pro, 1 hour, Biz and Ent, 1 second. Anything less then that just wouldn't be cached at edge. You cache it for longer though (let's say 24 hours) and call the cache purge api at any time though, might be easier to control it that way if you want to adjust how long it'll be cached for, like with the main page

RapidOP•1/15/24, 6:02 PM

I see, okay, thanks! I'll look through it first

RRapid I see, okay, thanks! I'll look through it first

Chaika•1/15/24, 6:06 PM

There are other potential options as well, could leverage KV and local cache in a Worker, could be kind of pricey though.
Which plan are you on?
There's a feedback loop there where the more requests you get, the more places your assets will be in cache, the lower latency, etc, but can't force it. Cloudflare wants only frequent assets in cache, there's a finite amount of space

RapidOP•1/15/24, 6:07 PM

Currently using pro plan

RapidOP•1/15/24, 6:08 PM

Can increase to business if it is needed for that

Chaika•1/15/24, 6:10 PM

As I was saying above Cache is per colo/datacenter. Cloudflare has points of presence (PoPs), identified by airport codes like ORD (Chicago), EWR (New York). Some of those PoPs are multi-colo (MCPs), and each with their own cache. There's some 200-300 PoPs or something and around 550 colos. If there was a global cache pull api that would be a ton of requests and 550 copies of each asset lol, that's why there are tools like tiered caching and such. Generally speaking though your visitors are not going to be that evenly spread, just going to be a few popular points most likely

CChaika As I was saying above Cache is per colo/datacenter. Cloudflare has points of pre...

RapidOP•1/15/24, 6:11 PM

Yeah, I'm just looking to make sure users get it almost instatnly from anywhere, but the main thing is google ofcourse

RapidOP•1/15/24, 6:12 PM

I just saw your description, ClickHouse and Cloudflare workers, what are you doing with ClickHouse if I may ask :)?

Chaika•1/15/24, 6:13 PM

You can use the CDN-Cache-Control response header to control Cloudflare's cache ttl and such, and then Cache-Control would just be for the browser:
https://developers.cloudflare.com/cache/concepts/cdn-cache-control/
ex, cache main page and all other pages for 24 hours, purge via API whenever needed. Cloudflare has an extension for Wordpress (APO) which does exactly that, not a bad pattern

CDN-Cache-Control · Cloudflare Cache (CDN) docs

CDN-Cache-Control is a response header field set on the origin to separately control the behavior of CDN caches from other intermediaries that might …

RRapid I just saw your description, ClickHouse and Cloudflare workers, what are you doi...

Chaika•1/15/24, 6:13 PM

Analytics and lots of data

Could move to off-topic to talk about that
Cloudflare also uses Clickhouse for all of its analytics, traffic graphs, Cloudflare Radar and such, it's really powerful

Phatso•1/15/24, 7:45 PM

Is there an HTTP API that lets me clear the cache for a specific route?

My content for this endpoint changes randomly - sometimes once a week, sometimes multiple times per day.
I want to clear the cache for that page after my site re-deploys

PPhatso Is there an HTTP API that lets me clear the cache for a specific route? My cont...

Chaika•1/15/24, 7:47 PM

https://developers.cloudflare.com/api/operations/zone-purge#purge-cached-content-by-url is the purge cache api.
Non-Enterprise can only do by exact url

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Chaika•1/15/24, 7:48 PM

otherwise you could just purge everything in that zone/website

CChaika https://developers.cloudflare.com/api/operations/zone-purge#purge-cached-content...

Phatso•1/15/24, 7:48 PM

Exactly what I was looking for, thank you

Chaika•1/15/24, 7:48 PM

no, not in the ruleset engine

Chaika•1/15/24, 7:49 PM

like, redirect http -> http, https -> https?

Chaika•1/15/24, 7:50 PM

if you have Always use HTTPS it would redirect everyone first/you could just assume everything -> https.
If you needed to keep it, you could just have two rules, one matching ssl and another without

Chaika•1/15/24, 7:52 PM

yea I don't think there's any easier way for that, if it was relative (i.e /contact-us/ to /contacts/), you could just have it be static and relative, but if it's to a different domain I think you're forced to put scheme.

Ashley•1/15/24, 8:12 PM

Is https://blog.cloudflare.com/secrets-store in beta yet, or any news on it? As the post was 7 months ago, and I've not seen anything mentioned since (but maybe I missed news)

The Cloudflare Blog

Announcing Cloudflare Secrets Store

Introducing Secrets Store by Cloudflare - the ultimate solution for managing your application secrets securely. Safeguard sensitive information, track access, and maintain ease of use.

villainy•1/15/24, 8:13 PM

am i missing something? when trying to launch area 1, i just get this prompt:

villainy•1/15/24, 8:13 PM

am i supposed to get an email

Vvillainy am i missing something? when trying to launch area 1, i just get this prompt:

Chaika•1/15/24, 8:14 PM

are you an area 1 customer? That's all enterprise stuff, if you wanted normal email routing stuff it's under Email tab in a zone/website

CChaika are you an area 1 customer? That's all enterprise stuff, if you wanted normal e...

villainy•1/15/24, 8:16 PM

so i need an enterprise account? yes i am exploring the email routing features, and wanted to look at the sec section

AAshley Is https://blog.cloudflare.com/secrets-store in beta yet, or any news on it? As ...

Chaika•1/15/24, 8:17 PM

Probably delayed by Code Orange if nothing else, some products just have that happen anyway

Chaika•1/15/24, 8:18 PM

It looks like Mia was a pm intern asking for interviews
general-discussions
nothing further I know of

Vvillainy so i need an enterprise account? yes i am exploring the email routing features, ...

Chaika•1/15/24, 8:19 PM

The marketing page is here: https://www.cloudflare.com/zero-trust/products/email-security/, can request a free trial/talk to sales

villainy•1/15/24, 8:19 PM

Thanks

Vvillainy so i need an enterprise account? yes i am exploring the email routing features, ...

dave•1/15/24, 9:07 PM

depending on what you want to do, Cloudflare Email Workers might be a better solution than area 1.

CChaika https://developers.cloudflare.com/api/operations/zone-purge#purge-cached-content...

Phatso•1/15/24, 9:26 PM

I don't see how you're supposed to provide a full URL.
The endpoint takes

tags

tags

hosts

hosts

, and

prefixes

prefixes

- where do I put my full URL?

PPhatso I don't see how you're supposed to provide a full URL. The endpoint takes `tags`...

Chaika•1/15/24, 9:27 PM

The API Docs aim to confuse, you can click on the little drop down next to the body, the last one is for urls

Chaika•1/15/24, 9:27 PM

if in doubt you can see what the dashboard does as well, it does the same api call/endpoint

Phatso•1/15/24, 9:27 PM

Got it! Thanks

PPhatso Got it! Thanks

Chaika•1/15/24, 9:28 PM

example request body:

{
  "files": [
    "https://cookies.com/gihgohgdf",
    "https://cookies.com/gihgohgdhi"
  ]
}

{
  "files": [
    "https://cookies.com/gihgohgdf",
    "https://cookies.com/gihgohgdhi"
  ]
}

CChaika example request body: ```json { "files": [ "https://cookies.com/gihgohgdf"...

Phatso•1/15/24, 9:57 PM

So I have to use my Global API Key? I can't use a scoped API Token for this endpoint?

Hiya, We are creating our caching system at the moment, I'm wondering, is there a way to tell cloud

Similar Threads

Similar Threads

Similar Threads