Send canonical link header instead of noindex header

Development
4

For the new default SiteSetting.allow_indexing_non_canonical_urls = false this is the way it’s implemented right now - and it stays this way:

  • header noindex
  • html link-tag canonical (might be ignored)

Without patch and SiteSetting.allow_indexing_non_canonical_urls = true

  • – no header –
  • html link-tag canonical

With patch and SiteSetting.allow_indexing_non_canonical_urls = true

  • header: Link: <https://forum.example.com/t/test-example/1234>; rel="canonical"
  • html link-tag canonical (might be ignored - but anyway same as header)

The whole idea behind this:
Set canonical as http-header to get the same benefit like the noindex http-header - namely faster crawling.
Thereby this might make noindex obsolet with its uncertain implications.

Another point on noindex vs. canonical:

  • noindex is more than a very strong signal to not put the page into search index.
    But with noindex the page content is still processed by Google Bot to extract links (there is the extra option nofollow to disable this).
  • canonical is a strong signal that the content to be crawled is on some other canonical url.
    In case Google Bot decides to accept this signal for one page, there is a big chance it does not process the page content at all – and only processes the canonical url.

This is a ‘thought experiment’. It’s nowhere implemented - and I never recommend to implement it:

  • header noindex
  • html meta-tag noindex (instead of: html link-tag canonical)

– OR –

  • – no header –
  • html meta-tag noindex

Why or why not to implement it like this?