# Zoekmachines nu geblokkeerd voor het indexeren van niet-canonieke pagina's
**URL:** https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985
**Category:** Announcements
**Tags:** seo
**Created:** [21 februari 2022 om 19:35 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985 "2022-02-21T19:35:37Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [21 februari 2022 om 19:35 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/1 "2022-02-21T19:35:37Z")
> ⚠ **Important**
>
> Following further investigation we decided to leave non canonical indexing enable, see more details at: [Search engines now blocked from indexing non-canonical pages - #30 by sam](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/30)
> **original announcement**
>
> Discourse will now reply with a `X-Robots-Tag: noindex` header when the requested page isn’t the canonical page for a resource.
>
> While Discourse uses an automatic scrolling design for both topic lists and topics, this isn’t what we show search engine crawlers, like GoogleBot. Search engines see paginated topics, with 20 posts in each page. However, since users can link to specific posts in their own posts and will do so using the `/t/title/topic_id/post_id` URL format, those will be picked by the crawlers and add duplicated content into your site search results and waste the precious and limited crawl budget your domain have.
>
> To alleviate this issue, our community of users suggested adding the `X-Robots-Tag: noindex` to URLs like post specific URLs, which we managed to expand to all non-canonical URLs in Discourse. This was released as a hidden site setting and disabled by default 3 months ago, during which we experimented having this header enabled in community sites as well as on [meta.discourse.org](http://meta.discourse.org).
>
> **Since results of this period are looking good so far, we just flipped this setting to be in effect by default.**
>
> If you for some reason don’t want this behavior on your instance you can still enable indexing of non-canonical pages by running `docker exec -i app rails runner "SiteSetting.allow_indexing_non_canonical_urls = true"` on your server.
>
> Don’t expect any drastic changes on crawling and search results overnight, but over the next months you should see a decrease of crawls and search results on post specific pages, which will result in more crawl time spent on your site new topics and on content that wasn’t yet indexed because of crawl budget constraints on your domain.
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [21 februari 2022 om 23:29 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/4 "2022-02-21T23:29:37Z")
### Author:  [@Krischan](https://meta.discourse.org/u/Krischan)
#### Post date: [22 februari 2022 om 10:54 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/11 "2022-02-22T10:54:12Z")
> [@j127](#):
>
> Internal links help tell search engines which pages on the site are important.
This is a crucial point. It’s one thing to get all of your pages indexed and another to get a relevant ranking for them. In my experience (with big publisher sites), smart internal linking is key to achieve this.
---
### Author:  [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [22 februari 2022 om 17:48 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/12 "2022-02-22T17:48:15Z")
I updated just this morning, do you recommend enable indexing of non-canonical pages with this?
> [@Falco](#):
>
> If you for some reason don’t want this behavior on your instance you can still enable indexing of non-canonical pages by running `docker exec -i app rails runner "SiteSetting.allow_index_in_robots_txt = true"` on your server.
I would not want to make my [indexing more worse](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/7).
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [22 februari 2022 om 17:57 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/13 "2022-02-22T17:57:11Z")
> [@arkklo](#):
>
> I assume this feature is already available by default as long as we update Discourse to v2.9?
For anyone that updates their site since the OP post date.
> [@j127](#):
>
> If they see a URL like `example.com/t/1234/5` but have already crawled it and knows that its `canonical` URL is `example.com/t/1234` , they probably aren’t going to waste their computing resources visiting the non-canonical version multiple times.
We have data that shows that the new header reduces crawl time on those pages, and they always had the canonical set.
> [@rrit](#):
>
> All deep-links from external domains to posts (e.g. `forum.example.com/t/example-topic/5/11` ) have a http-header `X-Robots-Tag: noindex` now! I suggest to remove this http-header again.
But those pages are not meant to be crawled anyway. The metadata with the URL is set on the topic level, we don’t want Google to crawl the post level as it’s duplicated content.
> [@rrit](#):
>
> I suggest for `` to never use an URL with a post anchor (the last number in `…/t/example-topic/1234/5` ) anywhere. Canonical URLs should always point to the topic url itself ( `…/t/example-topic/1234` ). I think it is already implemented like this.
Cool, so nothing need to change here.
> [@rrit](#):
>
> New suggestion for crawler view _only_ :
> Convert all internal links pointing to a post-URL ( `example.com/t/1234/5` ) to point to the corresponding topic-URL ( `example.com/t/1234` ) instead.
Doing that at runtime may be too CPU expensive, and saving two versions of every post will disk expensive.
> [@mstm](#):
>
> I updated just this morning, do you recommend enable indexing of non-canonical pages with this?
Our defaults are always what we recommend. However, we maintain and announce site settings so people can choose otherwise if they feel like a default isn’t ideal for their site.
---
### Author:  [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [22 februari 2022 om 18:11 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/14 "2022-02-22T18:11:02Z")
> [@Falco](#):
>
> Our defaults are always what we recommend.
Perfect then I will leave as recommended.
Thank you
---
### Author:  [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [22 februari 2022 om 18:23 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/15 "2022-02-22T18:23:33Z")
Last thing and then I don’t disturb anymore 😅
So could there be problems with sitemap\_recent.xml that contains such links?
`https://meta.discourse.org/t/category-moderator-improvements/158628?page=2`
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [22 februari 2022 om 18:29 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/16 "2022-02-22T18:29:44Z")
That example is a canonical page, so it isn’t affect in any way by the changes outlined in the OP.
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [22 februari 2022 om 20:19 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/17 "2022-02-22T20:19:55Z")
> [@Falco](#):
>
> > [@rrit](#):
> >
> > All deep-links from external domains to posts (e.g. `forum.example.com/t/example-topic/5/11` ) have a http-header `X-Robots-Tag: noindex` now! I suggest to remove this http-header again.
>
> But those pages are not meant to be crawled anyway. The metadata with the URL is set on the topic level, we don’t want Google to crawl the post level as it’s duplicated content.
I see a huge difference when there is an **external link** to a post-url.
```plaintext
# A:
External Domain
|
|--(link juice)--> post-url
|
|__/ crawling: \---> post-url not indexed and
\ header noindex / link-juice mostly gone
# B:
External Domain
|
|--(link juice)--> post-url
|
| __/ crawling: \__ |--> post-url not indexed
\ answer canonical / |--> topic-url indexed (anyway)
with link-juice transfer
```
We should bring this up on
> **[How to Submit a Question for Google SEO Office Hours | Google Search Central...](https://developers.google.com/search/help/office-hours?hl=en)**
>
> Google SEO Office Hours is where Googlers from the Search Quality team answer questions around Google Search, websites, and the things between. Learn how to submit a question and get tips on getting your question answered.
---
### Author:  [@Canapin](https://meta.discourse.org/u/Canapin)
#### Post date: [22 februari 2022 om 21:44 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/18 "2022-02-22T21:44:05Z")
> [@Falco](#):
>
> Don’t expect any drastic changes on crawling and search results overnight, but over the next months you should see a decrease of crawls and search results on post specific pages, which will result in more crawl time spent on your site new topics and on content that wasn’t yet indexed because of crawl budget constraints on your domain.
For neophytes like me regarding SEO, does it imply that it’s an SEO improvement that could potentially lead to a slight increase/benefit in Google search results?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [22 februari 2022 om 22:26 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/19 "2022-02-22T22:26:47Z")
Yes, that is the goal!
We tested the change in a tech news community over a few months, and we saw a large peak-to-peak increase in anon page views. Our end goal is _always_ to make all Discourse communities healthier in all fronts.
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [22 februari 2022 om 23:47 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/20 "2022-02-22T23:47:29Z")
> [@Falco](#):
>
> more crawl time spent on your site new topics and on content that wasn’t yet indexed because of crawl budget constraints on your domain
Is this effect visible in Google Search Console report ‘Settings’ → ‘Crawling’ → ‘[Crawl stats](https://search.google.com/search-console/not-verified?original_url=/search-console/settings/crawl-stats&hl=en)’ ?
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [23 februari 2022 om 23:00 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/21 "2022-02-23T23:00:10Z")
### Taking into consideration …
A. Decreasing crawls
> [@Falco](#):
>
> decrease of crawls and search results on post specific pages, which will result in more crawl time spent on your site new topics and on content that wasn’t yet indexed because of crawl budget constraints on your domain.
B. No two versions of content
> [@Falco](#):
>
> Doing [link conversion for crawler view] at runtime may be too CPU expensive, and saving two versions of every post will disk expensive.
C. Use `canonical` tag
> [@j127](#):
>
> The `canonical` tag is the way Google recommends dealing with that problem, and it appears to be working already. Doing weird things with canonical tags can lead to strange problems with Google […]
D. No `nofollow`
> [@Krischan](#):
>
> > [@j127](#):
> >
> > I don’t think internal `nofollow` is ideal. Internal links help tell search engines which pages on the site are important.
>
> This is a crucial point. It’s one thing to get all of your pages indexed and another to get a relevant ranking for them. In my experience (with big publisher sites), smart internal linking is key to achieve this.
E. No `noindex`
> [@j127](#):
>
> […] a `noindex` mistake could be difficult to recover from.
> [@rrit](#):
>
> I see a huge difference when there is an **external link** to a post-url [concerning link-juice and `no-index`].
### … and having internal links at …
> [@rrit](#):
>
> Locations where such links to posts are found:
>
> - manually added links in user content
> - automatically generated links in
> - quotes
> - first post of topic: “inbound tracked links” from other topics
> - first post of topic: “selected answer”
> - first post of topic - topic map open: “topic links”/“liked links”
### … I suggest the follwoing implementation to get the best compromise:
1. Don’t add http-header `X-Robots-Tag: noindex`.
– taking into account [E] –
2. Keep `canonical` tags always pointing to the topic-url.
– decreasing crawls [A] and considering [C] –
3. _For crawler view only:_ Convert automatically generated links to always link to topic-url instead of post-url - for all links in first post of topic “inbound tracked links from other topics" and “topic map open: topic link/liked links”.
– decreasing crawls [A] and considering [D], but willfully disregarding [B] –
On [B]: CPU expenses are for crawler-visits only and consist of doing a regex-replace to cut off the last number of internal urls ending in two numbers e.g. `…/t/example-topic/1234/5` → `…/t/example-topic/1234` in the confined borders of first post of topic “inbound tracked links from other topics” and “topic map open” only.
4. _for all views:_ add internal `nofollow` to quotes and manually added links in user content.
– decreasing crawls [A] and considering [B], but slightly disregarding [D] –
On [D]: important links are already automatically duplicated to first topic in "topic map open: topic link/liked links”-section [see 3.] and most quotes stay inside the topic itself anyway.
* * *
### Some idea on internal links
Google says [How to Specify a Canonical with rel="canonical" and Other Methods | Google Search Central | Documentation | Google for Developers](https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls?hl=en#best-practices)
> [@](#):
>
> For all canonicalization methods, follow these general guidelines:
>
> - Link to the canonical URL rather than a duplicate URL, when linking within your site. Linking consistently to the URL that you consider to be canonical helps Google understand your preference.
And Google says [SEO Link Best Practices for Google | Google Search Central | Documentation | Google for Developers](https://developers.google.com/search/docs/advanced/guidelines/links-crawlable?hl=en#use-proper-a-tags)
> [@](#):
>
> ## Use proper `` tags
>
> **Can follow:**
>
> - 👍 ``
> - 👍``
>
> **Can’t follow:**
>
> - 👎 ``
> - 👎 ``
> - 👎 ``
So Discourse might set internal links like this:
```xml
…
```
For Google the link goes straight to the canonical topic-url `…/1234` - and Google does not get to know about the post-url `…/1234/5` from this link-syntax.
For user-navigation some additional JavaScript in the Ember-app will do the trick:
e.g. replace `href` with `routerLink`.
---
### Author:  [@SethWilliams](https://meta.discourse.org/u/SethWilliams)
#### Post date: [24 februari 2022 om 17:02 UTC](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985/22 "2022-02-24T17:02:46Z")
Looks like a great improvement! Thanks for making this happen @Falco and Discourse team!
[Next page](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985.md?page=2)