After the latest Google Panda update, I found myself on the unhappy end of a 15% drop in visitors via Google, which led me to do some research on what exactly Google has indexed for my site.
According to a quick search using “site:discuss.howtogeek.com/users/” I noticed that all of the user profiles are being indexed in Google, despite having almost no useful information on any of them. So on my very small forum that’s 4,310 additional URLs that are indexed in Google despite having very little quality content on them.
I noticed this setting, which allows me to entirely disallow the forum from Google, but I think it would be better to just have a way to exclude user profiles in the robots.txt.
The ideal scenario would be if I could just edit the robots.txt in the settings, of course.
Good catch. Web crawlers will see almost nothing there. My page as crawlers see it:
Do we even need a setting for this? We could just add a rule to the robots.txt to disallow indexing of user pages. I don’t think any Discourse install would want them to be indexed.
