I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

Interesting question. I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite. I also don’t know, but I assume, that any “behaving” bots would observe both nofollow, noindex in page heads and an all inclusive disallow in the robots.txt but m…

[תמונה] Mittineague: I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite One of the most frequent use cases for this is to keep a site out of search engines before it is live to the general public, without having to require everyon…

We are using our discourse site as a internal knowledge base for our company ( discourse.example.com ). We do not want search engines to crawl this website whatsoever. The three options I see in settings are: [תמונה] What security settings should we change in addition to disabling the “allow index…

Every decent search engine honors robots.txt so that should be enough. If telling them nicely doesn’t work then the only way to keep them out is making your forum private, or blocking certain IP addresses.

[תמונה] supernaturally: We are using our discourse site as a internal knowledge base for our company I do the exact same thing. We did two things to keep our site secure: Require Login. Not everyone at the “company” (university in my case) should have access, only certain staff, so we res…

[תמונה] jomaxro: 0 access through the firewa Excellent info - thank you, @jomaxro ! We would love to do #2, but our small company works out of a co-working space (we don’t own the network). Do you have any suggestions for a workaround, or do you think that #1 would be sufficient?

#1 is absolutely sufficient. Any unauthenticated viewer (like Google) would see the following page (customizable). There is no way to access any content, and thus no way to crawl anything. Adding NOINDEX should be respected, but if not the worst case is hitting a login screen: [תמונה] As for #…

So if I’m storing a lot of sensitive information on our discourse forum, what do you suggest we do to make it as secure as possible? (we don’t want intruders that find intranet.ourcompany.com , to be able to get in and read sensitive product and strategy info). Should I just remove our CNAME and ha…

Login required is a pretty massive lock-down. So go with login required, must approve users, invite only. With that setup, only accounts that the site staff send invites to can see anything on the site.

[תמונה] supernaturally: but security is not my strong suit I strongly recommend you hire someone who knows. Answers to your question have already been explained on the thread.

How to disable indexing by crawlers

תמיכה

codinghorror (Jeff Atwood) 9 בפברואר,‏ 2016,‏ 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

MegaIndex bot did about 4,000 pageviews on one day

נושא		תגובות	צפיות
Robots.txt to completely block all indexing of private site Support	4	918	6 במרץ,‏ 2020
Why there are lots of Disallow rule in robots.txt? Support	34	4687	22 בדצמבר,‏ 2020
Temporarily block Google crawler while setting up live site? Support	3	493	6 בינואר,‏ 2020
Pages listed in the robots.txt are crawled and indexed by Google Support	18	3391	30 ביולי,‏ 2019
Option to index topics Support	7	588	31 בדצמבר,‏ 2022

How to disable indexing by crawlers

נושאים קשורים