Search engines now blocked from indexing non-canonical pages

Announcements
4

TL;DR: Don’t block non-canonical pages - just point them to a correct url via <link rel="canonical" … > - that’s what it’s made for.

This feature might harm the SEO link-building in the long run:
All deep-links to answers inside topics are on noindex pages now! Does Google like this?

Actually a canonical tag always pointing to the topic url - even for pages deep-linking on an answer - should perfectly do the job – without adding X-Robots-Tag: noindex:
On first crawl of a deep-linking answer page Google recognizes that the page url (answer inside topic) does not fit the canonical-url and then decides to only crawl the canonical-url (topic).

May we add <a rel="nofollow" …> to all links doing this topic-answer deep-linking? Edit: no, see Search engines now blocked from indexing non-canonical pages - #9 by j127
Thereby we might save even more of this precious and limited crawl budget of search engines:
the search engine would neither extract the link in the first place nor do a call to the url. As calling the url results in a response with a X-Robots-Tag: noindex http-header causing the response to be ‘trashed’ by adding the url to the search engines internal ‘noindex’-list.

Some more savings on crawl budget with nofollow added to RSS-feed urls:

5 Likes