Suggested Topics not shown to web spiders

(Love Chopra ) #1

I was looking at the Topic Page on Lynx Text Browser to simulate how Google bot is viewing this page and what all links Google bot finds for crawling and interlinking purpose. Please look at the details below:

Suggested Topics links are not getting rendered in text browser.


  1. Take any discourse url, I took a url from
    Poll: What's your preferred mobile OS? - tech - Demo

  2. When we access it in any web browser, suggested topic are there:

  3. But when we access the same page in Lynx text browser Suggested Topics are missing

As you can see there are no interlinks of other related topics on this page and I think if this can be added this will give advantage over SEO, I have also checked stackover and in stackover flow all the related links are perfectly getting rendered on text browser as well as on web browser.

(Dean Taylor) #2

Without a user account (which is what Googlebot sees) the list of suggested topics is just randomly generated, which is almost pointless to Google as it the links have little relevance to this topic.

If anything the only additional links to topics / posts to include would be the “links to the posts in this topic”. Where this topic / posts have been referenced by posts in another topic - you would normally see in the right-hand sidebar combined with outgoing links.

(Love Chopra ) #3

@DeanMarkTaylor but there is no interlinking structure on this page if it does not have any linked topic. If we look at any standard content websites like any wordpress blog or if we take reference of our most favorite stackoverflow there is a well defined and strategic interlinking structure on each of the question pages, please have a look at below screen shot:

(Sam Saffron) #4

Incorrect, different beasts.

StackOverflow uses a heuristic to find related topics based on text in the question. We do not.

We should though display “incoming” links, outgoing are covered.

(Love Chopra ) #5

Yeah its a different one, I placed that just for a reference purpose, just to put light on all the high traffic website uses the similar methodology for their main content pages, infact wordpress blogs are also designed in a similar way.
In discourse we only have topic pages as our main content pages which can drive high quality organic traffic.

I believe if somehow we could add links to other related topics on each of the topic pages then it will improve overall interlinking of discourse and will result in better SEO.

People uses wordpress just because it perform best in SEO as compared to other blogging platform similarly if we could build that capability to discourse which is already 100X better then wordpress in term of product but I fee it still minutely lagging in SEO.

(Mittineague) #6

You’ve been paying attention to other than Google.
Cross-linking can help in getting pages indexed,

If I do a
search I get “About 15,600” results. I’d say Discourse is doing fine in terms of SEO

(Love Chopra ) #7

@Mittineague interlining of pages is a very important metrics for SEO.
Discourse is weak in interlinking, please look at the below screenshot generated from moz, it is clearly saying Total Internal Links are only 6,287 which should have be quite higher as you mentioned it has 15,000+ url indexed in google.

I strongly recommend we should have some strong mechanism for better interlinking on topic pages like all the standard blogs has.

(Mittineague) #8

You know this because you work for Google?
Do you know of a Google source reference?

(Love Chopra ) #9

I don’t work for google but I am talking about this because I looked into data of many high traffic content websites and all have similar paterns

(Mittineague) #10

Well, it’s up to the Discourse Team whether or not they would invest the time into it or accept a PR, but IMHO conjecture isn’t a strong enough motivation for potential SEO benefit.

(William) #11


Has there been any changes on that matter (new feature/plugin that allows it)?
It would be great at least to be able to choose if those links are displayed to robots or not.
They already are there for humans so I don’t see the point in specifically hiding them for robots.

(Sam Saffron) #12

Suggested topics are mostly random for anon, they do not relate to the actual topic. linked topics on the other hand carry a lot of weight, we should confirm we at least render them. (for the case of incoming links)

(Jeff Atwood) #13

@techapj can you add to your list to confirm linked topics are rendered under posts for web spiders, I agree with this.

(William) #14

Thanks for the replies
However I still don’t understand the reason to show suggested topics to humans and specifically hide it from spiders, even if generated randomly?
Am I missing something?

(Sam Saffron) #15

It produces no value for spiders and confuses ranking

(William) #16

Is there any proof of this?
Has there been a case study?

I would say that internal linking increases rankings and that consistent behaviour for humans and robots is a plus.


(Sam Saffron) #17

Thing is, the list is random and changes each time the page reloads, so it is inconsistent by definition

(William) #18

How about not making them random?
It could be for example the list of suggested topics the user gets when opening the topic

(Jeff Atwood) #19

If you wish to build a plugin that behaves that way, you are free to do so. Discourse is 100% open source software so you can make it into whatever you want.

However, a radical redesign of the current functionality is not on our roadmap.

(Mittineague) #20

IMHO especially if it is “writing for search bots” instead of “writing for humans”.

I use Suggested almost exclusively for navigating to topics of interest. I find it works extremely well for my use. I would be disappointed if it was changed to something that was a type of “Top” list. When I want Top, I go to Top.