Google blocked by Cloudflare Edge despite whitelist? Can't make sense of logs!
I'm having an issue where Google's crawler is being blocked by Cloudflare's edge, even though we specifically have a whitelisting rule that skips all security rules.
Mainly, I can't make sense of the log explorer entry. The log explorer says "CacheStatus=unknown" which would suggest it's not being served from cache. However, the origin is not delivering any response, so the 403 response was clearly sent by Cloudflare.
Second, the security rule triggered is our whitelisting rule. The rule basically makes it so that all custom rules are skipped for verified search engines. I've attached a screenshot of the rule, and below is an example of a block event in our logs.
I'd really appreciate help with figuring out what is causing the 403 response to occur. Thank you so much!
log
CacheCacheStatus
unknown
CacheReserveUsed
false
CacheResponseStatus
403
CacheTieredFill
false
ClientCountry
us
ClientIP
66.249.68.35
ClientRequestHost
www.worldhistory.org
ClientRequestMethod
GET
ClientRequestPath
/Kublai_Khan/
ClientRequestURI
/Kublai_Khan/
ClientXRequestedWith
-
Date
2025-07-29
EdgeEndTimestamp
2025-07-29T04:24:39Z
EdgeRequestHost
www.worldhistory.org
EdgeResponseStatus
403
EdgeStartTimestamp
2025-07-29T04:24:39Z
OriginResponseStatus
0
OriginResponseTime
0
RayID
9669d91038c3d44d
RequestHeaders
{}
SecurityRuleDescription
Whitelist: Search Engines
SecurityRuleID
bc909ecfc0dd4018827b43a8872d5e66
SecurityRuleIDs
bc909ecfc0dd4018827b43a8872d5e66
WorkerStatus
unknown
WorkerSubrequest
false
Mainly, I can't make sense of the log explorer entry. The log explorer says "CacheStatus=unknown" which would suggest it's not being served from cache. However, the origin is not delivering any response, so the 403 response was clearly sent by Cloudflare.
Second, the security rule triggered is our whitelisting rule. The rule basically makes it so that all custom rules are skipped for verified search engines. I've attached a screenshot of the rule, and below is an example of a block event in our logs.
I'd really appreciate help with figuring out what is causing the 403 response to occur. Thank you so much!
log
CacheCacheStatus
unknown
CacheReserveUsed
false
CacheResponseStatus
403
CacheTieredFill
false
ClientCountry
us
ClientIP
66.249.68.35
ClientRequestHost
www.worldhistory.org
ClientRequestMethod
GET
ClientRequestPath
/Kublai_Khan/
ClientRequestURI
/Kublai_Khan/
ClientXRequestedWith
-
Date
2025-07-29
EdgeEndTimestamp
2025-07-29T04:24:39Z
EdgeRequestHost
www.worldhistory.org
EdgeResponseStatus
403
EdgeStartTimestamp
2025-07-29T04:24:39Z
OriginResponseStatus
0
OriginResponseTime
0
RayID
9669d91038c3d44d
RequestHeaders
{}
SecurityRuleDescription
Whitelist: Search Engines
SecurityRuleID
bc909ecfc0dd4018827b43a8872d5e66
SecurityRuleIDs
bc909ecfc0dd4018827b43a8872d5e66
WorkerStatus
unknown
WorkerSubrequest
false
