Noindex - Follow in Tags Meta robots - Avoid duplicate content

(Veer) #1

Tags are a big necessity for my forum, i want tags to either not indexed in robots.txt, or it will be best if it have Noindex - follow in its Header meta robots.
How can i achieve it ?
It can be very very benefitial to avoid any Google panda or penguin penalty.
@denharsh

  • Also how to put sitemap link in Robots.txt ?
The «Tagging» core functionality adds multiple pages with the same human content (but different to Google) to SERP
How to Noindex thin & duplicate pages in Discourse?
The «Tagging» core functionality adds multiple pages with the same human content (but different to Google) to SERP
(Sam Saffron) #2

robots.txt is not really customizable without a plugin. However, I agree the various /tags/ routes should probably not be indexed by google, seems like an inefficient way to reach content.

we should look into this, also I am not totally against adding sitemap support at some point.

How to Noindex thin & duplicate pages in Discourse?