So there have been a number of conversations around the robots.txt file, and how some need / don’t need to edit it based on their specific use-case. What I haven’t seen is exactly where to edit it (knowing that after an upgrade one would need to re-edit the file again). I see in /var/www/discourse/app/views there is a robots.txt directory with some ruby files but not an actual robots.txt file. So where exactly does this file reside?

I just searched for Google documentation regarding “thin links” but came up empty. Got a link to the Google documentation handy?

And you’ve gotten “Thin content with little or no added value” message(s) on your Manual Actions Report page for the “latest” page? Anyway, getting a bit off-topic, sorry. You can write a plugin to add rules to the robots.txt file https://meta.discourse.org/t/how-to-add-to-robots-txt-host/59718/4?u=mittineague

If this is something that would be of benefit to many people, consider making a PR against core to add it either as a default, or behind a site setting.

[image] jerdog: It would benefit everyone if we at least could get Latest added to the robots.txt file I would be extremely against this cause how would google then be able to find new content? Latest is very much required. @codinghorror what I totally support here is adding noindex, follow to meta tags for /latest and even category topic lists, it will clean up a bunch of messy results. Thoughts?

The problem is that we store “latest” content in the search index which is pointless cause we never want people to land on “latest”. The tricky thing is that any mucking around here can rock the boat and this is not a boat we want to rock. I have seen rare reports of this kind of stuff with user profiles not have noindex in meta tags, but not the reverse.

I’m gonna need to see some screenshot Google Search Console proof to believe this is a problem, because we’ve checked our console pretty closely for years.

As requested, here are screenshots from one site: [58%20PM] [18%20PM]

This is completely unrelated and we should probably add noindex tags on user pages. This was raised before wrt Bing.

Needing to edit robots.txt file - where is it?

Product Support

codinghorror (Jeff Atwood) August 6, 2018, 12:39am 9

I am not really following? What’s the problem? Our meta.discourse.org Google Search Console has been running for years, we check it every few months in quite some detail, and I’ve never seen any problems with this reported?

Topic		Replies	Views
Why there are lots of Disallow rule in robots.txt? Product Support	34	5021	December 22, 2020
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Product Support	18	437	December 18, 2024
Excluding user profiles in robots.txt (or allow edit of file) Feature	4	2636	May 24, 2014
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3631	July 30, 2019
Google notification to remove "noindex" statements from robots.txt Product Support	7	2638	July 30, 2019

Needing to edit robots.txt file - where is it?

Related topics