Google Cache / Google Crawling does not include Suggested Topics


(Carl Hendy) #1

Hi all,

Long time lurker and fan of Discourse. Yes, a huge fan so before I go down the SEO path I want to be clear this is not a criticism more of a suggestion. I have searched for this topic but could not find it discuss previously.

I noticed that when Google caches a thread that only parts of the thread are cached with that being the main content.

If you look at the cache of this URL: https://community.coinbase.com/t/how-can-i-get-a-profit-and-loss-statement-showing-my-gains-over-the-past-2-years/9484

Google Cache: http://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fcommunity.coinbase.com%2Ft%2Fhow-can-i-get-a-profit-and-loss-statement-showing-my-gains-over-the-past-2-years%2F9484&rlz=1C5CHFA_enAU736AU736&oq=cache%3Ahttps%3A%2F%2Fcommunity.coinbase.com%2Ft%2Fhow-can-i-get-a-profit-and-loss-statement-showing-my-gains-over-the-past-2-years%2F9484&aqs=chrome..69i57j69i58.1103j0j4&sourceid=chrome&ie=UTF-8

Google Cache Text Version: webcache.googleusercontent.com/search?q=cache:https://community.coinbase.com/t/how-can-i-get-a-profit-and-loss-statement-showing-my-gains-over-the-past-2-years/9484&num=1&rlz=1C5CHFA_enAU736AU736&strip=1&vwsrc=0

You will noticed that Google is not seeing the “Suggested Topics” or rendering all other links / content on the page.

Would it be possible to get the [Suggested Topics] to be rendered also as this will allow Google and other bots to follow those links and crawl / discover related / new pages for indexing?

In this example: Categories - The SitePoint Forums

Google cache: Categories - The SitePoint Forums

We can see that Google is not caching the [Latest] sidebar links which have all the new threads linked to. As this page would be an authorative landing page there would be much value for a website to gain from allowing those [Latest] links to be crawlable to search engine bots.

The basic of web crawling discovery and PageRank scoring are built on search engines (Google) to be able to find links on pages and crawl those links to new pages. This crawling also allows Google to create an “internal PageRank score” which helps pages perform better in SEO, improving visbiolity and traffic.

I have no knowlege as to how much work this would be to allow these other parts of the template to be cached but would beneficial to every website running Discourse.

Any questions please let me know.

Thanks
Carl

p.s had to delete some WWW references as new user only allowed to post 2 links


(Jeff Atwood) #2

Suggested topics are random so it would be a random walk. @techapj added linked topics to the crawler view a while ago which was I think more important.


(Carl Hendy) #3

Why not make suggested topics more relevant and useful for the visitor?


(Jeff Atwood) #4

They are relevant you just need to be logged in for them to be contextual to you, and Google is not a logged in user, nor will it ever be.


(Christoph) #5

So suggested topics are based only on the user’s history, they have nothing to do with the topic undee which they’re displayed?