Hi. I’ve seen the discussion at: [image] Needing to edit robots.txt file - where is it? Support You are completely hijacking a topic here that is asking for removal of items from the Google index. Feel free to open a dedicated topic to discuss this. There are changes we do support like having a reasonable HTML off version of the user summary page… however you are doing is totally derailing this topic. We have survived fine for the last 5 years without indexing user pages, lots of users DO NOT want their forum user page indexed by Google. Tons of profiles are thin and pointless. Turning th… but I did not find a clear cut answer to this question. How can I enable a robots.txt for a Discourse site to indicate that it should not be indexed by any crawlers. This is regarding a private site and there’s no reason for it to have any presence in any search engines. Thanks!

Ah, missed that one down there. Perfect. Worked like a charm. Thanks!

Also, make sure you have the login required setting turned on by rerunning the wizard at yoursite.com . That will also shut out search engines.

The site has always required login, and does not permit user self-registration. However, it has still been showing up in search engines – just with no content indexed since it couldn’t log in. My hope is that with robots.txt blocking explicitly, any mention will cease. Thanks.

Robots.txt to completely block all indexing of private site

Product Support

jomaxro (Joshua Rosenfeld) March 5, 2020, 5:46pm 2

See the second to last reply in that topic: Needing to edit robots.txt file - where is it? - #42 by Osama.

Topic		Replies	Views
How to disable indexing by crawlers Product Support	15	5488	January 18, 2017
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3632	July 30, 2019
Excluding user profiles in robots.txt (or allow edit of file) Feature	4	2636	May 24, 2014
Need to find a solution to block indexing topics from a category Product Support	2	178	November 27, 2024
Malformed robots.txt causing issues with indexing Product Support	7	2036	September 14, 2019

Robots.txt to completely block all indexing of private site

Related topics