I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

Interesting question. I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite. I also don’t know, but I assume, that any “behaving” bots would observe both nofollow, noindex in page heads and an all inclusive disallow in the robots.txt but m…

[image] Mittineague: I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite One of the most frequent use cases for this is to keep a site out of search engines before it is live to the general public, without having to require everyon…

We are using our discourse site as a internal knowledge base for our company ( discourse.example.com ). We do not want search engines to crawl this website whatsoever. The three options I see in settings are: [image] What security settings should we change in addition to disabling the “allow index…

Every decent search engine honors robots.txt so that should be enough. If telling them nicely doesn’t work then the only way to keep them out is making your forum private, or blocking certain IP addresses.

[image] supernaturally: We are using our discourse site as a internal knowledge base for our company I do the exact same thing. We did two things to keep our site secure: Require Login. Not everyone at the “company” (university in my case) should have access, only certain staff, so we res…

[image] jomaxro: 0 access through the firewa Excellent info - thank you, @jomaxro ! We would love to do #2, but our small company works out of a co-working space (we don’t own the network). Do you have any suggestions for a workaround, or do you think that #1 would be sufficient?

#1 is absolutely sufficient. Any unauthenticated viewer (like Google) would see the following page (customizable). There is no way to access any content, and thus no way to crawl anything. Adding NOINDEX should be respected, but if not the worst case is hitting a login screen: [image] As for #…

So if I’m storing a lot of sensitive information on our discourse forum, what do you suggest we do to make it as secure as possible? (we don’t want intruders that find intranet.ourcompany.com , to be able to get in and read sensitive product and strategy info). Should I just remove our CNAME and ha…

Login required is a pretty massive lock-down. So go with login required, must approve users, invite only. With that setup, only accounts that the site staff send invites to can see anything on the site.

[image] supernaturally: but security is not my strong suit I strongly recommend you hire someone who knows. Answers to your question have already been explained on the thread.

How to disable indexing by crawlers

Support

codinghorror (Jeff Atwood) February 9, 2016, 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

MegaIndex bot did about 4,000 pageviews on one day

Topic		Replies	Views
Robots.txt to completely block all indexing of private site Support	4	918	March 6, 2020
Why there are lots of Disallow rule in robots.txt? Support	34	4712	December 22, 2020
Temporarily block Google crawler while setting up live site? Support	3	493	January 6, 2020
Pages listed in the robots.txt are crawled and indexed by Google Support	18	3392	July 30, 2019
Option to index topics Support	7	589	December 31, 2022

How to disable indexing by crawlers

Related topics