Semantic microdata and more content for crawlers


(Erick Guan) #1

Breadcrumb is one of many semantic markup described in http://schema.org. Search engines rely on these markups to improve the display of search result.

This feature request is to add more things for crawler.

  • More microdata can be added into topic view, lists, category list and faq page. Maybe it can help for displaying topic list in the search engines.

Beyond that, add some content for crawler in some views.

  • Add category description in the category list.
  • Add link to category list in topic list page. There is no link to get there.

BTW, can I add 360Spider to crawler ua list? It’s the second search engine in China now. (appr. Baidu 56% - 360 30%. bye to the third search engine for the last 10 :cold_sweat: )


(Jeff Atwood) #2

Sure add 360 UA that is fine.


(Erick Guan) #3

Does this looks fine to you? @codinghorror

https://github.com/fantasticfears/discourse/commit/4ede84758e5d4536b6fc50c4440bce374fa6e115


(Jeff Atwood) #4

I would not link Top, that is entirely duplicate links from the crawler’s perspective. The rest seems fine.


(Erick Guan) #5

What about <meta name="robots" content="nofollow"> for top page? Then the top page can be accessible for noscript, eg Lynx.


(Jeff Atwood) #6

I seriously doubt anyone is using those crawler-only pages.