We continued analyzing issues following this change and decided to roll it back per:
The goal behind it was to limit crawl budget by Google being spent scanning non canonical topic links.
Since this change was applied we rolled out 2 fixes that made the change unnecessary.
-
Topic RSS feeds are no longer followed, links in the RSS feeds are not followed. Eg:
https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985.rss -
Post RSS feeds now contain canonical links. Eg:
https://meta.discourse.org/posts.rss
Combined these two changes mean crawlers no longer discover a large amount on non-canonical links on Discourse sites.
The frees search budget and makes the site setting no longer a requirement. Site operators are still free to experiment with it, however it is disabled by default.