This stuff need extremely careful consideration
- do we want users entering discourse sites on a category filter page?
- do we want users entering discourse sites on a category filter page on page 100
- do we want users to get a hit on a “list” style page in the expense of hitting the right topic?
- do we want users entering on top page (probably yes)
- is a site map desirable to increase crawling efficiency ?
Having the same canonical for all the pages on the list stuff for non topics heavily deemphasises them as search results, something that is desirable
I wonder if we should even allow robots to index any of the filters except for latest.
You can get to every topic on the site through latest, the fact we allow all this slice and diced crawling does make crawling activity much less effective, as Google keeps on rediscovering the same content over and over
We simply need to analyze our logs first and see how big the problem is, there is huge appeal in decreasing crawling load and increasing crawling efficiency it makes all sites faster and better
But we need to be ultra careful here not to cause any unwanted side effects that take months to rectify