Search engines now blocked from indexing non-canonical pages

[rrit (Ayke)]

Some more information on noindex from the Google docs:

---

See Crawl Budget Management | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers

• Consolidate duplicate content. Eliminate duplicate content to focus crawling on unique content rather than unique URLs.
• Block crawling of URLs that you don’t want indexed. Some pages might be important to users, but you don’t want them to appear in Search results. For example, infinite scrolling pages that duplicate information on linked pages, or differently sorted versions of the same page. If you can’t consolidate them as described in the first bullet, block these unimportant (for search) pages using robots.txt or the URL Parameters tool (for duplicate content reached by URL parameters).

Don’t use noindex , as Google will still request, but then drop the page when it sees the noindex tag, wasting crawling time. Don’t use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don’t want Google to crawl at all. Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.

---

See How to Specify a Canonical with rel="canonical" and Other Methods | Google Search Central  |  Documentation  |  Google for Developers

Specify a canonical page

To specify a canonical URL for duplicate URLs or similar pages, choose one of the following methods. […]

Method and description

• rel=canonical <link> tag
• rel=canonical HTTP header

[…]

See How to Specify a Canonical with rel="canonical" and Other Methods | Google Search Central  |  Documentation  |  Google for Developers

For all canonicalization methods, follow these general guidelines:

• Don’t use the robots.txt file for canonicalization purposes.
• Don’t use noindex as a means to prevent selection of a canonical page. This directive is intended to exclude the page from the index, not to manage the choice of a canonical page.
• Link to the canonical URL rather than a duplicate URL, when linking within your site. Linking consistently to the URL that you consider to be canonical helps Google understand your preference.

[…]