A discussion about Negative SEO due to excessive category pages being indexed on a Discourse site is underway. veer points out that on meta.discourse.org, 1670 category pages are indexed, but on their own Discourse site with 50 categories, over 23,000 pages are indexed, which is a dangerous signal for SEO due to content duplicacy.
Stranik suggests that possible disallows in the robots.txt file could be the cause. Mittineague questions the assumption that the Google search modifier “site:” behaves in a way that limits results to specific folders.
codinghorror discovers that the issue is caused by weird query strings, such as no_subcategories, slow_platform, and per_page, which result in multiple versions of the same category page being indexed. They propose that a canonical meta-tag should be added to category pages to prevent this issue.
techAPJ reveals that the canonical meta-tag was previously included but was regressed 4 months ago. They fix the issue and add a test case to prevent future regression. The fix is later backported to the beta and stable branches.
If you look for total category pages indexed of meta.discourse.org with following query in google
site:meta.discourse.org/c/
you will find 1670 pages index. Similarly i have a discourse site with 50 categories
while here more than 23 thousand pages are indexed under category. (A very very dangerous signal for seo because of content duplicacy)
It should be fixed otherwise google will penalise a discourse website rankings.
Look at the following screenshot how duplicate pages are created and indexed.
I think it is more correct to look directly into the search index. I don’t know if there is such a mechanism in the Google search engine. In Yandex there is a detailed report: duplicates, poor content, all types of errors, are excluded and the included page, graphics crawls, etc. According to the results, a lot of duplicate pages.
We used to include canonical meta tag on /c/plugin page but (I think) this was regressed 4 months ago when we introduced “Default Topic List” category setting.
This should now be fixed, and a test case is added to prevent future regression.
