I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

Interesting question. I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite. I also don’t know, but I assume, that any “behaving” bots would observe both nofollow, noindex in page heads and an all inclusive disallow in the robots.txt but m…

[изображение] Mittineague: I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite One of the most frequent use cases for this is to keep a site out of search engines before it is live to the general public, without having to require e…

We are using our discourse site as a internal knowledge base for our company ( discourse.example.com ). We do not want search engines to crawl this website whatsoever. The three options I see in settings are: [изображение] What security settings should we change in addition to disabling the “allow…

Every decent search engine honors robots.txt so that should be enough. If telling them nicely doesn’t work then the only way to keep them out is making your forum private, or blocking certain IP addresses.

[изображение] supernaturally: We are using our discourse site as a internal knowledge base for our company I do the exact same thing. We did two things to keep our site secure: Require Login. Not everyone at the “company” (university in my case) should have access, only certain staff, so …

[изображение] jomaxro: 0 access through the firewa Excellent info - thank you, @jomaxro ! We would love to do #2, but our small company works out of a co-working space (we don’t own the network). Do you have any suggestions for a workaround, or do you think that #1 would be sufficient?

#1 is absolutely sufficient. Any unauthenticated viewer (like Google) would see the following page (customizable). There is no way to access any content, and thus no way to crawl anything. Adding NOINDEX should be respected, but if not the worst case is hitting a login screen: [изображение] As…

So if I’m storing a lot of sensitive information on our discourse forum, what do you suggest we do to make it as secure as possible? (we don’t want intruders that find intranet.ourcompany.com , to be able to get in and read sensitive product and strategy info). Should I just remove our CNAME and ha…

Login required is a pretty massive lock-down. So go with login required, must approve users, invite only. With that setup, only accounts that the site staff send invites to can see anything on the site.

[изображение] supernaturally: but security is not my strong suit I strongly recommend you hire someone who knows. Answers to your question have already been explained on the thread.

How to disable indexing by crawlers

Поддержка

codinghorror (Jeff Atwood) 09.Февраль.2016 11:14:26 3

There is a boolean site setting for this, search for “index” or “robots”.

7 лайков

MegaIndex bot did about 4,000 pageviews on one day

Тема		Ответов	Просм.
Robots.txt to completely block all indexing of private site Support	5	882	05.04.2020
Why there are lots of Disallow rule in robots.txt? Support	34	4588	22.12.2020
Temporarily block Google crawler while setting up live site? Support	4	473	05.02.2020
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3303	30.07.2019
Option to index topics Support	8	560	31.12.2022

How to disable indexing by crawlers

Связанные темы