Search engines now blocked from indexing non-canonical pages

[rrit (Ayke)]

Some more information on noindex from the Google docs:

---

See Crawl Budget Management For Large Sites | Google Search Central  |  Documentation  |  Google Developers

• Consolidate duplicate content. Eliminate duplicate content to focus crawling on unique content rather than unique URLs.
• Block crawling of URLs that you don’t want indexed. Some pages might be important to users, but you don’t want them to appear in Search results. For example, infinite scrolling pages that duplicate information on linked pages, or differently sorted versions of the same page. If you can’t consolidate them as described in the first bullet, block these unimportant (for search) pages using robots.txt or the URL Parameters tool (for duplicate content reached by URL parameters).

Don’t use noindex , as Google will still request, but then drop the page when it sees the noindex tag, wasting crawling time. Don’t use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don’t want Google to crawl at all. Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.

---

See Consolidate Duplicate URLs with Canonicals | Google Search Central  |  Documentation  |  Google Developers

Specify a canonical page

To specify a canonical URL for duplicate URLs or similar pages, choose one of the following methods. […]

Method and description

• rel=canonical <link> tag
• rel=canonical HTTP header

[…]

See Consolidate Duplicate URLs with Canonicals | Google Search Central  |  Documentation  |  Google Developers

For all canonicalization methods, follow these general guidelines:

• Don’t use the robots.txt file for canonicalization purposes.
• Don’t use noindex as a means to prevent selection of a canonical page. This directive is intended to exclude the page from the index, not to manage the choice of a canonical page.
• Link to the canonical URL rather than a duplicate URL, when linking within your site. Linking consistently to the URL that you consider to be canonical helps Google understand your preference.

[…]

[j127 (Josh)]

@Falco

docker exec -i app rails runner "SiteSetting.allow_index_in_robots_txt = true"

This command doesn’t seem to work. I updated a smaller Discourse site today to test it, ran the command, and still see the noindex headers.

---

Edit: I’m not sure how that setting works, but I don’t see it in the SiteSettings, at least from the frontend (as admin) in the browser console:

var d = Discourse.SiteSettings;
document.body.innerHTML = `<pre>${JSON.stringify(d, null, 4)}</pre>`;

It looks like that setting is for robots.txt, not noindex. Wouldn’t that already be true on most Discourse sites?

[Falco]

Oh sorry the correct is SiteSetting.allow_indexing_non_canonical_urls. Fixed it on the OP.

[sam (Sam Saffron)]

We continued analyzing issues following this change and decided to roll it back per:

FEATURE: enable canonical url indexing

The goal behind it was to limit crawl budget by Google being spent scanning non canonical topic links.

Since this change was applied we rolled out 2 fixes that made the change unnecessary.

1. Topic RSS feeds are no longer followed, links in the RSS feeds are not followed. Eg: https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985.rss

2. Post RSS feeds now contain canonical links. Eg: https://meta.discourse.org/posts.rss

Combined these two changes mean crawlers no longer discover a large amount on non-canonical links on Discourse sites.

The frees search budget and makes the site setting no longer a requirement. Site operators are still free to experiment with it, however it is disabled by default.