Hi. I’ve seen the discussion at: [imagem] Needing to edit robots.txt file - where is it? Support You are completely hijacking a topic here that is asking for removal of items from the Google index. Feel free to open a dedicated topic to discuss this. There are changes we do support like having a reasonable HTML off version of the user summary page… however you are doing is totally derailing this topic. We have survived fine for the last 5 years without indexing user pages, lots of users DO NOT want their forum user page indexed by Google. Tons of profiles are thin and pointless. Turning th… but I did not find a clear cut answer to this question. How can I enable a robots.txt for a Discourse site to indicate that it should not be indexed by any crawlers. This is regarding a private site and there’s no reason for it to have any presence in any search engines. Thanks!

See the second to last reply in that topic: Needing to edit robots.txt file - where is it? - #42 by Osama.

Ah, missed that one down there. Perfect. Worked like a charm. Thanks!

Also, make sure you have the login required setting turned on by rerunning the wizard at yoursite.com . That will also shut out search engines.

The site has always required login, and does not permit user self-registration. However, it has still been showing up in search engines – just with no content indexed since it couldn’t log in. My hope is that with robots.txt blocking explicitly, any mention will cease. Thanks.

Robots.txt para bloquear completamente a indexação de site privado

Suporte ao Produto

jomaxro (Joshua Rosenfeld) 5 Março , 2020 17:46 2

Veja a segunda a última resposta nesse tópico: Needing to edit robots.txt file - where is it? - #42 by Osama.

Tópico		Respostas	Vistas
How to disable indexing by crawlers Product Support	15	5468	18 Janeiro , 2017
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3592	30 Julho , 2019
Excluding user profiles in robots.txt (or allow edit of file) Feature	4	2630	24 Maio , 2014
Need to find a solution to block indexing topics from a category Product Support	2	170	27 Novembro , 2024
Malformed robots.txt causing issues with indexing Product Support	7	2020	14 Setembro , 2019

Robots.txt para bloquear completamente a indexação de site privado

Related topics