Cloudflare is allowing search engines to index my website even if I have blocked it in robots.txt
My robots.txt file is
User-agent: *
Disallow: /
I enabled the cloudflare AI blocker and the new robots.txt generated is the following. As you can see the first directive is allowing search engines from indexing my website even if the last directive is denying it. I tested it and google is using the first directive and scanning my site. How can I prevent it and properly block search engines from indexing my website?
User-Agent: *
Content-signal: search=yes,ai-train=no
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: *
Disallow: /
User-agent: *
Disallow: /
I enabled the cloudflare AI blocker and the new robots.txt generated is the following. As you can see the first directive is allowing search engines from indexing my website even if the last directive is denying it. I tested it and google is using the first directive and scanning my site. How can I prevent it and properly block search engines from indexing my website?
BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes,ai-train=no
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
END Cloudflare Managed Content
User-agent: *
Disallow: /
