robots.txt
Hi - I'm moving a previous wordpress site to a static site using pages and a cloudflare custom domain. I've added a robots.txt file to the files uploaded - when I visit mysite.com/robots.txt or mysite.com/anystring it's showing the index page - how do I stop that behaviour - I'd rather the user/crawler was served a 404 for mysite.com/anystring than the home page?
7 Replies
Serving Pages · Cloudflare Pages docs
Cloudflare Pages includes a number of defaults for serving your Pages sites. This page details some of those decisions, so you can understand how …
thanks that's really useful - I've added a 404 and it's all behaving better - I'm still getting the index page served on the custom domain site.com when I try and view site.com/robots.txt .... (I have added robots.txt to the root folder I upload to deploy and can see it there in the uploaded files) how can I get pages to not deisplay the index page if googlebot comes looking for robots.txt?
I've carried on digging around - am I going to need something like proxy flare to serve the robots.txt? https://developers.cloudflare.com/pages/platform/functions/plugins/community-plugins/
Community Plugins · Cloudflare Pages docs
The following are some of the community-maintained Pages Plugins. If you have created a Pages Plugin and would like to share it with developers, …
Nope, it should just serve it
Can you take a screenshot of the "Assets uploaded" tab in the dashboard?
I can give account ID in case it's something with the custom domain that's causing it - didn't want to share url publicy
Hmm, yeah do you want to DM me your account ID + domain
sure
resolved - thanks very much for your help - for anyone in the future reading this double check response with curl https://www.yoursite.com/robots.txt