Search engines now blocked from indexing non-canonical pages

announcements
30

We continued analyzing issues following this change and decided to roll it back per:

The goal behind it was to limit crawl budget by Google being spent scanning non canonical topic links.

Since this change was applied we rolled out 2 fixes that made the change unnecessary.

  1. Topic RSS feeds are no longer followed, links in the RSS feeds are not followed. Eg: https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985.rss

  2. Post RSS feeds now contain canonical links. Eg: https://meta.discourse.org/posts.rss

Combined these two changes mean crawlers no longer discover a large amount on non-canonical links on Discourse sites.

The frees search budget and makes the site setting no longer a requirement. Site operators are still free to experiment with it, however it is disabled by default.

12 Likes