I’m wondering if there’s a way to remove canonical links entirely from a Discourse forum. I’m referring to the /2, /3, /4, etc extensions that show up for each topic URL as a user scrolls down the page. I would like to have every reply within a topic simply refer to the original URL (not redirecting visitors to the original URL, but removing those paths completely, so they don’t exist). Originally, I thought this was a cool feature of Discourse, but as I’ve been running a similar forum on my site over the past year with NodeBB (which uses the same canonical link feature for each reply), we’ve discovered that this functionality can be catastrophic for the SEO of a public forum. Why? Because even though these /2, /3, /4 URLs are canonical links, Google will eventually crawl and index all of them. This means that every new reply within a topic can show up in Google search results, and since they’re basically duplicate versions of the original topic URL, these extra indexed pages rarely get visits and when they do, visitors don’t stay on the site for more than a couple of seconds. When Google indexes a lot of extra pages and these pages don’t get much activity, it tells Google that the domain as a whole has a lot of low-quality URLs and it hurts the domain health as a whole (as it did in our case). We lost about 40% of our traffic since launching our public forum, and a big contributing factor was these additional URLs (over 30,000 low-quality URLs were added to our sitemap over the course of 12 months, simply because of the replies that were left in each topic). Now, if you’re running a private forum, none of this matters, because a private forum doesn’t benefit from any kind of SEO since the whole forum is hidden from the internet anyway. But if you ARE trying to run a public forum and your objective is to pick up organic search traffic, these additional URL strings can have a huge detrimental impact on the overall health of your site. So, I’m wondering, is there any conceivable way to tweak the settings or create a plugin that will tell a Discourse forum NOT to create these additional URLs for each reply within a topic? I’m considering migrating our forum from NodeBB over to Discourse, but this will only make sense if there’s a way to make our Discourse forum not create these additional URLs.

[image] RGJ: [image] SethWilliams: over 30,000 low-quality URLs were added to our sitemap over the course of 12 months Are you sure about this? I have never seen a post number in a site map. Thanks for questioning this. “Sitemap” was probably the wrong word to use. What I meant was that these numbered posts were being crawled and indexed by Google and showing up as individual pages in Google Analytics and Search Console, resulting in A LOT of low-quality pages on our domain. If these numbers simply weren’t added to every single reply, Google would only be able to see the original post URL.

[image] SethWilliams: If these numbers simply weren’t added to every single reply, Google would only be able to see the original post URL. Yeah, and if Google would be honoring those rel="canonical" meta tags (which they invented!) that were put in there specifically to prevent Google from doing this, it would be not an issue and we would still be able to link to a specific post at the same time.

Sounds good. It would be ideal if it’s off by default, because it’s not inconceivable that it could cause topic pages to disappear from Google or other search engines. I’m not sure if it was already mentioned, but another way to fix it without noindex might be to use URL fragments for the posts, since those shouldn’t get counted as separate pages. /t/slug/id#13

[image] j127: I’m not sure if it was already mentioned, but another way to fix it without noindex might be to use URL fragments for the posts, since those shouldn’t get counted as separate pages. Thanks for the suggestion. I’d like to think that would work… but seeing as how the original “correct” method isn’t even working, it makes me skeptical that anything will solve the problem other than just eliminating the numbered replies altogether. Of course, I’m not saying ALL Discourse users should stop using them. Heck, if it weren’t for Google being dumb (and/or if we intended for our forum to be private and not public), I would be all for it… but just having the option to turn off the automatic numbering of replies would be a huge help to those who run public forums and care about the overall SEO health of their domain.

[image] SethWilliams: but seeing as how the original “correct” method isn’t even working, it makes me skeptical that anything will solve the problem other than just eliminating the numbered replies altogether. That would remove the ability to link to specific posts though. There would be no way to link to post #789 in a 1,000-post topic, and it would be annoying for users to have to scroll that far. It’s strange. I searched Google to see if the post ID URLs were getting indexed on my forums, and only the canonical URLs are showing on all the topics I checked. I do see it on another large Discourse site though. It also appears on this topic. [Google query] I ran a diff on the two responses like this: curl -s https://meta.discourse.org/t/removing-the-2-3-4-etc-links-for-each-reply-within-a-topic-url/209648 > 1.html curl -s https://meta.discourse.org/t/removing-the-2-3-4-etc-links-for-each-reply-within-a-topic-url/209648/8 > 2.html vim -d 1.html 2.html One difference that stands out is that the article:published_time is different though they should probably be the same, because the pages are otherwise almost identical. I wonder if that metatag could make Google override the canonical URL. A Google employee says that canonical URLs can be ignored in certain cases over here. Also, is the ignore_canonical tag unique to Discourse or is there a chance that Google might be following it? I saw it in the HTML.

[image] j127: That would remove the ability to link to specific posts though. There would be no way to link to post #789 in a 1,000-post topic, and it would be annoying for users to have to scroll that far. It could be slightly more annoying, but if the page=2, page=3, etc URLs still work (which doesn’t really create an SEO problem like the numbered replies do), you could at least link a person to the correct page within a conversation. This would get them most of the way there, provided they’re willing to scroll a little bit. One forum that works this way is BiggerPockets. Their replies do not have individually numbered URLs, but the topics do have numbered pages, like this: Kids throwing rocks at windows nearly everyday. Won't stop.Kids throwing rocks at windows nearly everyday. Won't stop. (take note of the URL as you’re scrolling through each topic and page). Their forum has always been a major component of what makes the site so special and successful from an SEO standpoint, so it’s a pretty good example of what works. [image] j127: One difference that stands out is that the article:published_time is different though they should probably be the same, because the pages are otherwise almost identical. I wonder if that metatag could make Google override the canonical URL. A Google employee says that canonical URLs can be ignored in certain cases over here . Interesting. I have no idea if that’s causing the problem or not, but I can see how the inconsistency might confuse Google into ignoring the canonical URL. Even so, with the nature of how forums work, unless you wanted to completely remove the dates and timelines of each topic, you couldn’t eliminate this, could you? Accounting for the dates and times of each post and reply is sort of an integral part of how forums work.

[image] j127: they should probably be the same, because the pages are otherwise almost identical. No, /8 is not the same as the topic. /8 points to the 8th post and the timestamp corresponds to that of the 8th post. If you compare the ?page=2 variant to the actual post it links to then you will get the same timestamps. For instance: wget -q -O - https://meta.discourse.org/t/topic-list-previews-legacy/101646/959|grep published_ti wget -q -O - https://meta.discourse.org/t/topic-list-previews-legacy/101646/?page=2|grep published_ti [image] j127: Also, is the ignore_canonical tag unique to Discourse Looks like it: Incorrect or failing oneboxes for links to other discourse instances - #14 by techAPJ

[image] SethWilliams: Accounting for the dates and times of each post and reply is sort of an integral part of how forums work. I’m not saying to remove time information, but just that it would be better to only send the machine-readable timestamp for the top post. From the perspective of ranking a page in search results, a forum topic is basically an article (top post) with a bunch of comments on it. It doesn’t matter to a search engine when the comments were made. Edit: another way of passing the date to Google for a comment (as opposed to the entire page) is schema.org markup. [image] RGJ: No, /8 is not the same as the topic. /8 points to the 8th post and the timestamp corresponds to that of the 8th post. Sure, /8 points to the 8th post, but from a bot’s perspective and from Google’s perspective, it’s the exact same content and URL. If you want Google to know that /8 should be treated the exact same way as the topic in the search results, then the site probably shouldn’t send an intentional signal that they are different. Only the human user needs to know that the timestamps are different, and that information is printed in the text on the page. If someone at Google has to make decisions about when to override site-defined canonical URLs, one of those exceptions could be something like “two different timestamps in the intentional metadata means different pages – therefore override the canonical URL.” It’s often hard for programmers to think of all the edge cases unless they have experience with encountering that thing, so it might be inconceivable to the Google programmers that identical pages could have two different timestamps, even though it’s easy for Discourse users to understand why that might happen. I used to work at a company where part of my job was to get sites unbanned from Google. (They weren’t doing anything shady, but there were just technical problems.) Since no one knew exactly how Google’s ranking tech works, and it changes regularly, the starting place was to try to think like a Search engineer and remove anything that could possibly be ambiguous or confusing to machines. I could never say exactly which thing worked, but it always worked after some time of systematically fixing things like that.

[image] Falco: to add a X-Robots-Tag: noindex header to the response payload of those pages. This is in. If you want to enable this experimental feature, you need to flip the value to the hidden site setting SiteSetting.allow_indexing_non_canonical_urls. Please share the results with us.

[image] j127: I’m not saying to remove time information, but just that it would be better to only send the machine-readable timestamp for the top post. From the perspective of ranking a page in search results, a forum topic is basically an article (top post) with a bunch of comments on it. It doesn’t matter to a search engine when the comments were made. Makes perfect sense to me. [image] j127: Sure, /8 points to the 8th post, but from a bot’s perspective and from Google’s perspective, it’s the exact same content and URL. If you want Google to know that /8 should be treated the exact same way as the topic in the search results, then the site probably shouldn’t send an intentional signal that they are different. Only the human user needs to know that the timestamps are different, and that information is printed in the text on the page. If someone at Google has to make decisions about when to override site-defined canonical URLs, one of those exceptions could be something like “two different timestamps in the intentional metadata means different pages – therefore override the canonical URL.” It’s often hard for programmers to think of all the edge cases unless they have experience with encountering that thing, so it might be inconceivable to the Google programmers that identical pages could have two different timestamps, even though it’s easy for Discourse users to understand why that might happen. Yes, yes, and yes. Well articulated.

Removing the /2, /3, /4, etc links for each reply within a topic URL

Development

David_Ghost October 13, 2024, 1:51pm 35

I think the problem here is the crawl budget.

Topic		Replies	Views
Search engines now blocked from indexing non-canonical pages Announcements seo	23	4886	March 15, 2022
Sitelinks in Google disappearing Community Building	26	1878	January 27, 2023
Why isn't Google Indexing Discourse? SEO concerns Product Support seo	31	6437	June 1, 2024
Adding Canonical Redirects for SEO Optimization Product Support	23	7770	October 1, 2015
Pagination URL scheme not passed through when topic is renamed Feature	21	4362	May 20, 2015

Removing the /2, /3, /4, etc links for each reply within a topic URL

Related topics