I found the reason for the internal error but the cause is that the object sent is somehow malformed

I found the reason for the internal error but the cause is that the object sent is somehow malformed (without the internal error it might give you more context).

Oom The close datacenter to my location is hit in 3 request but far away datacenter ...

Isaac McFadyen•9/27/22, 7:16 PM

As has been said before, cache is per-datacenter.

Isaac McFadyen•9/27/22, 7:16 PM

Hitting a datacenter will not cache it in another data center.

Isaac McFadyen•9/27/22, 7:17 PM

Are those 4 requests hitting the other datacenter for sure?

Erisa•9/27/22, 7:24 PM

i curled it and it said MISS, i curled it again right after and it says HIT

Unsmart•9/27/22, 7:24 PM

Cache is per datacenter so yes a different datacenter will miss.

Erisa•9/27/22, 7:24 PM

its now cached in LHR (London) and will return HIT for anyone accessing Cloudflare through their London datacenter

Erisa•9/27/22, 7:24 PM

but not others

Oom https://tools.keycdn.com/performance

Isaac McFadyen•9/27/22, 7:25 PM

If you take a look at the Ray ID for these requests you'll notice they are actually hitting different datacenters.

Isaac McFadyen•9/27/22, 7:25 PM

For example, when I tested first it hit MUC in Frankfurt (a MISS)

Isaac McFadyen•9/27/22, 7:25 PM

Then when I tested again it was a MISS in Frankfurt again, but this time FRA

Isaac McFadyen•9/27/22, 7:25 PM

So it's actually caching: just that tool is sending to different data centers in the region.

Isaac McFadyen•9/27/22, 7:32 PM

Can you check the actual RayID? That Colo field is likely their colo, not the Cloudflare colo.

Oom Click to see attachment

Isaac McFadyen•9/27/22, 7:36 PM

See here. Their request at CDG is hitting LHR: so the colo field they provide is not the Cloudflare colo it's hitting.

Isaac McFadyen•9/27/22, 7:36 PM

If you were to look, you would see that requests that hit the same colo always cache.

Oom Like how can I test actually

Isaac McFadyen•9/27/22, 7:38 PM

There's really nothing to test. It will cache in every colo that is hit more than once.

Isaac McFadyen•9/27/22, 7:38 PM

That's just how cache works.

Erisa•9/27/22, 7:46 PM

wait

Erisa•9/27/22, 7:47 PM

i'll reply properly, i misread at first

VVitamin Protein btw why tired cache dont work for custom domain ? I mean they r not using worker...

Erisa•9/27/22, 7:48 PM

R2's origin is a Worker that runs outside of your account.
Tiered Caching relies on knowing the closest datacenter to the origin.
The closest datacenter to a worker is.. nowhere, so it doesn't work. There is no real origin in the equation.

The R2 folks will have to add explicit support to make Tiered Caching work.

VVitamin Protein okay but we were told R2 bucket created in the nearest dc from where the request...

Erisa•9/27/22, 7:53 PM

we were told R2 bucket created in the nearest dc from where the request was originated

This is true!

so I thought if r2 is in eu then may be make lhr as upstream cache

If they added explicit support for Tiered Caching within R2, this is likely what it would do. The issue (from my outside view) is that it still goes through a Worker on their side which runs against a technical limitation in Tiered Caching where it won't work with a Worker. It is hoped this will be resolved in future.

The reason normal cache works is because your zone cache runs before their Worker does.

Erisa•9/27/22, 7:53 PM

its a worker on Cloudflare's internal account yeah

Erisa•9/27/22, 7:54 PM

your cache can run in front but the origin stuff that tiered caching does breaks

EErisa > we were told R2 bucket created in the nearest dc from where the request was or...

Erisa•9/27/22, 7:55 PM

Since this is a technical limitation I doubt anyone will be able to work around it very easily, and a hacky solution may stop working at any time so it probably isn't worth trying.

At the end of the day, it just isn't available right now. If you absolutely require Tiered Caching for your usecase then you're welcome to look into other object storage providers. If that isn't feasible for you, then you will have to live without this feature until it's fixed I'm afraid.