Managed Rules block Google Crawls

We've seen a significant increase in Google encountering 403 permission denied errors. They've got up into the thousands on our Google Search Console. I investigated this in detail and found that it's Cloudflare's Managed Rules, in particular the "Fake Google User Agent" that is blocking genuine Google crawls. Has anyone else encountered this? Is there any solution apart from turning off that managed rule?

ThamisOP•7/21/25, 8:46 AM

Right, but it's definitely the real request from Google.

ThamisOP•7/21/25, 8:47 AM

The timestamp matches exactly.

ThamisOP•7/21/25, 8:47 AM

How can you tell that it's a workers subrequest?

Laudian•7/21/25, 8:48 AM

If you're sure it's matching actual Google requests, you're probably using Signed Exchanges or something like that.

Laudian•7/21/25, 8:48 AM

Though those shouldn't be blocked.

ThamisOP•7/21/25, 8:49 AM

We do have TollBit that runs workers, so it could be that. But why would a subrequest affect the final result that Google sees?

ThamisOP•7/21/25, 8:50 AM

BTW, thanks for jumping in to help on this so quickly.

Laudian•7/21/25, 8:51 AM

Do you use Automatic Signed Exchanges? You can find it in the CF dashboard under Speed -> Optimization -> Other

ThamisOP•7/21/25, 8:51 AM

Yes, that is turned on.

Laudian•7/21/25, 8:52 AM

Whenever Google makes a request, Cloudflare will make a Worker subrequest for the Signed Exchange. This subrequest should be exempt from blocks if I remember right.

ThamisOP•7/21/25, 8:53 AM

Yes that would make sense that it's exempted. But we're definitely returning a 403 to Google.

ThamisOP•7/21/25, 8:55 AM

Based on our log explorer, no 403 was ever sent...

Laudian•7/21/25, 8:56 AM

I don't think Signed Exchanges would show there.

ThamisOP•7/21/25, 8:56 AM

Would you recommend turning off SGX?

ThamisOP•7/21/25, 8:57 AM

Or turning off the fake agent?

Laudian•7/21/25, 8:58 AM

You can disable the fake user agent rule until this has been fixed, unless you're having specific issues with fake Googlebots.

ThamisOP•7/21/25, 8:59 AM

Right, we've done that. How would you go about trying to fix this issue?

Laudian•7/21/25, 8:59 AM

Can you also create a support ticket and share the number?

Laudian•7/21/25, 9:00 AM

And can you see in your logs when this started?

ThamisOP•7/21/25, 9:00 AM

I can see in my Google Search Console when it started, yes.

Laudian•7/21/25, 9:01 AM

When was that?

ThamisOP•7/21/25, 9:02 AM

22nd of April

Laudian•7/21/25, 9:02 AM

Wow, it's been a while :NotLikeThis:

ThamisOP•7/21/25, 9:03 AM

Yup. We've been struggling to figure out why for a long time.

ThamisOP•7/21/25, 9:03 AM

I only now managed to get to the bottom of it.

ThamisOP•7/21/25, 9:03 AM

Here's the blocking event.

firewall-event-9614675167779357.json1.28KB

ThamisOP•7/21/25, 9:03 AM

Well, one of them.

ThamisOP•7/21/25, 9:07 AM

Right, I submitted a support case ticket. Case ID: 01659769

ThamisOP•7/21/25, 9:10 AM

The start of the issue does coincide with us deploying TollBit I think.

TThamis The start of the issue does coincide with us deploying TollBit I think.

Laudian•7/21/25, 9:11 AM

How exactly is TollBit deployed?

ThamisOP•7/21/25, 9:11 AM

It's a Cloudflare worker.

ThamisOP•7/21/25, 9:12 AM

https://docs.tollbit.com/publisher-bot-deterrence#cloud-flare

TollBit Documentation

TollBit Documentation and API reference

Learn how to onboard and implement TollBit for your application.

Laudian•7/21/25, 9:15 AM

Can you try a WAF skip rule for something like "cf.worker.upstream_zone equals yourdomain" -> skip everything?

ThamisOP•7/21/25, 9:15 AM

Sure. What does that do?

Laudian•7/21/25, 9:15 AM

It allows Worker from your own account

Laudian•7/21/25, 9:15 AM

to bypass any other rules

ThamisOP•7/21/25, 9:16 AM

With or without www?

Laudian•7/21/25, 9:17 AM

(cf.worker.upstream_zone eq "example.com")

(cf.worker.upstream_zone eq "example.com")

Laudian•7/21/25, 9:17 AM

The name of the Cloudflare zone, so without www

ThamisOP•7/21/25, 9:17 AM

Ok. Done!

ThamisOP•7/21/25, 9:18 AM

Like this?

Laudian•7/21/25, 9:18 AM

yes.

ThamisOP•7/21/25, 9:18 AM

Right, I've set that up!

ThamisOP•7/21/25, 9:19 AM

Question: If all traffic triggers that worker, would this basically whitelist everything?

ThamisOP•7/21/25, 9:20 AM

I'm not entirely sure how workers work. Do they run in parallel to the main request, or are they in between the request and the server?

Laudian•7/21/25, 9:21 AM

The WAF sits in front of every request. The initial request makes it through the WAF, but then the Worker subrequest will need to pass the WAF again.

Laudian•7/21/25, 9:21 AM

If the initial traffic made it through the WAF, you don't want to block the Worker subrequest.

ThamisOP•7/21/25, 9:22 AM

Ah that makes sense! So if the WAF blocks it in the first place, the worker won't get triggered. But otherwise the worker triggers another WAF event which might cause a block. Got it.

Laudian•7/21/25, 9:25 AM

Do you now still see issues if you enable the fake Google bot rule again?

ThamisOP•7/21/25, 9:26 AM

I'll have to wait for another crawl.

Managed Rules block Google Crawls

Similar Threads

Similar Threads

Similar Threads