To completely block all indexing of a private Discourse site, L30110 was looking for a solution to enable robots.txt. jomaxro pointed them to a relevant discussion, where the solution was found. Adding the robots.txt file successfully blocked crawlers. Additionally, riking suggested ensuring the login required setting is turned on, but L30110 mentioned this was already the case. Despite this, the site was still showing up in search engines, but with no indexed content. The robots.txt block is expected to remove any mention of the site from search engines.
Hi. I’ve seen the discussion at:
but I did not find a clear cut answer to this question. How can I enable a robots.txt for a Discourse site to indicate that it should not be indexed by any crawlers. This is regarding a private site and there’s no reason for it to have any presence in any search engines. Thanks!
See the second to last reply in that topic: Needing to edit robots.txt file - where is it? - #42 by Osama.
Ah, missed that one down there. Perfect. Worked like a charm. Thanks!
Also, make sure you have the login required setting turned on by rerunning the wizard at yoursite.com . That will also shut out search engines.
The site has always required login, and does not permit user self-registration. However, it has still been showing up in search engines – just with no content indexed since it couldn’t log in. My hope is that with robots.txt blocking explicitly, any mention will cease. Thanks.