I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search engines, where’s the best place to block countries, is it before Docker NAT or after, inside of Docker? I’d like to block entire countries. It seems it’s cheapest to drop connections with iptables before they hit the container, but I am not sure if that’s a good practice.

Interesting question. I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite. I also don’t know, but I assume, that any “behaving” bots would observe both nofollow, noindex in page heads and an all inclusive disallow in the robots.txt but maybe some do one and not the other? Sorry, I can’t offer anything in terms of a server doing the blocking. But maybe you could require registration and manual approval to keep the not-wanteds from accessing the site. They could still hit the site, but all they’d get is a “sorry”.

[image] Mittineague: I think I’ve never heard of anyone wanting to block search engines from an entire site. It’s usually the opposite One of the most frequent use cases for this is to keep a site out of search engines before it is live to the general public, without having to require everyone to login in order to be able to see any content. The setting is Security → allow index in robots txt

We are using our discourse site as a internal knowledge base for our company (discourse.example.com). We do not want search engines to crawl this website whatsoever. The three options I see in settings are: [image] What security settings should we change in addition to disabling the “allow index in robots txt”?

Every decent search engine honors robots.txt so that should be enough. If telling them nicely doesn’t work then the only way to keep them out is making your forum private, or blocking certain IP addresses.

[image] supernaturally: We are using our discourse site as a internal knowledge base for our company I do the exact same thing. We did two things to keep our site secure: Require Login. Not everyone at the “company” (university in my case) should have access, only certain staff, so we restricted access this way. No port 80 access through the firewall. You cannot access Discourse (or anything on the specific server for that matter) from outside the company network. This includes search engines (who are already prevented by #1, but I digress). If an employee needs access offsite, they use the VPN.

[image] jomaxro: 0 access through the firewa Excellent info - thank you, @jomaxro ! We would love to do #2, but our small company works out of a co-working space (we don’t own the network). Do you have any suggestions for a workaround, or do you think that #1 would be sufficient?

#1 is absolutely sufficient. Any unauthenticated viewer (like Google) would see the following page (customizable). There is no way to access any content, and thus no way to crawl anything. Adding NOINDEX should be respected, but if not the worst case is hitting a login screen: [image] As for #2, it’s not in place to prevent indexing of Discourse, but for overall security of the server (there are other apps besides Discourse). Company network security policy is not to open the firewall for a server unless there is a compelling reason to do so. The only use of the server is by students/employees, all of whom have VPN credentials, so the server doesn’t get firewall access.

So if I’m storing a lot of sensitive information on our discourse forum, what do you suggest we do to make it as secure as possible? (we don’t want intruders that find intranet.ourcompany.com, to be able to get in and read sensitive product and strategy info). Should I just remove our CNAME and have everyone direct to the IP address of the hosted server? I know these questions a bit novice, but security is not my strong suit.

Login required is a pretty massive lock-down. So go with login required, must approve users, invite only. With that setup, only accounts that the site staff send invites to can see anything on the site.

[image] supernaturally: but security is not my strong suit I strongly recommend you hire someone who knows. Answers to your question have already been explained on the thread.

How to disable indexing by crawlers

Product Support

codinghorror (Jeff Atwood) February 9, 2016, 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

MegaIndex bot did about 4,000 pageviews on one day

Topic		Replies	Views
Robots.txt to completely block all indexing of private site Product Support	4	1020	March 6, 2020
Why there are lots of Disallow rule in robots.txt? Product Support	34	5019	December 22, 2020
Temporarily block Google crawler while setting up live site? Product Support	3	562	January 6, 2020
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3626	July 30, 2019
Option to index topics Product Support	7	745	December 31, 2022

How to disable indexing by crawlers

Related topics