So I just noticed that Discourse serves a non-JS version of the site to Google…
I’ve been using JS to set the sites title, meta description, canonical, etc and seems google is getting really good at rendering client-side javascript ( I’m referring to my regular site, not discourse )
Is there a way to disable this and not give google bot a specific version?
…Also, discourse has a do-follow link there for itself “Powered by Discourse.” - Is there anything I can do about that?
Edit 1 : I should also add, I don’t mind having a Discourse link in there, perhaps no-follow , or do-follow but not site-wide
Probably a good idea to explain why this is necessary. The out-of-box behavior within Discourse works fine for the majority, based on your account it’s your choice with the above which is creating the conflict.
When I say I’m using JavaScript for the title, canonical and meta-description I mean for my regular website, not the discourse setup. I was just trying to convey my experiences with googlebot and JS
I tried adjusting the robots file, that didn’t seem to serve the JavaScript version to google
Edit: Im also starting to grasp how much of discourse is an app , client-side - very cool stuff . Still would be interesting to see what the google bot can do with it. I imagine the algorithm would really like the clean setup.
Right. It’s all an app. And there is little chance that Google could make sense of it, so there would need to be a second app, just for Google, which is why it sends the static version.
Just to scratch the itch, is there a way to test this? - Im just curious what the site will look like if we give google a try. They’ve started to really open up to the idea of client-side JS as of late. SEO wise at least it makes sense to try
The non-js version serves hyperlinks in the same color as the text. So you cant tell what a link is ( if you’re google bot )
The source code is available, so you’d just change something there. I am pretty familiar with the code, but have no idea where to look.
I think that if that were true there would already be an answer, as a bunch of people who know more about Discourse than I do and more about SEO than either of us care pretty deeply about this matter.
Do you really think that the Google bot is color blind and looks at text color to decide what is a link? And it isn’t able to read the HTML to see what is an href? That seems improbable.
I was thinking of starting a new topic here on meta to discuss this (Im not familiar enough with this pull requests/ commit jargon on github)
Trust me, Im shocked too… I also wouldn’t call myself a beginner when it comes to SEO, I’ve been on and off being involved for well over a decade. Whats going on here are negative ranking factors for sure. It may not hurt “LetsEncypts” SEO but smaller guys, for sure.
I think the majority of discourse users may share your mentality as far as SEO. Not giving it much thought. There’s a ton of hidden potential here.
Yes they do know its a link, they also know its dark just like the text and users won’t like that. Let’s not forget, google is now ranking us on what they see in the non-JS version.
In regards to the “Powered by Discourse” external link:
I remember watching a video with John Mueller/Matt Cutts - the Google SEO guys and they mentioned hidden internal links ( due to menu’s, JS etc ) dont really do damage. -
He also mentioned though, if its an external link , that you only show googlebot and not end-users, that it could be an issue.
I feel like im in the twilight zone to be honest. I guarantee you, once this is fixed , all Discourse forums will see a huge jump in Google organic traffic
It’s really easy to make claims like the above, particularly when by your own admission you don’t understand the inner workings nor design decisions which led to the current state.
Realistically speaking though you can’t guarantee anything here, and unless you can convince someone with the requisite skills to alter the code it’s going to remain hypothetical.
If you have a budget to explore this the marketplace might be a good starting point.
I have a very broad idea, the app is Javascipt heavy and historically Google has had issues with JS. So you guys went with the simple non-JS we have now. My argument is to at least TRY and see what google renders with JS enabled.
Side note:
It seems the hyperlinks in the body being the same color as the text was in a somewhat recent update. Older versions of my page on google show the links are blue.
I know this is quite an older topic now, but as I have currently myself an interest in all the Discourse and SEO topic, and Nader seems to be around, let’s revive it.
I don’t like very much either that Discourse serves something different specifically to Google than what is served to everybody else. I always heard this was the 101 of things to NOT do, because Google doesn’t like that at all.
Wasn’t one of the problem, or even the main one, Discourse was seen as slow to load with JS enabled?
This has been worked around by serving a static version to Google and by displaying the “loading” animation to show something is appearing pretty quickly. “Something” yes, but not the information the user searched for and why he clicked on the link. This takes time to show. And by serving the JS version, Google would know it. While it is currently somewhat tricked to not know how long it takes.
Or am I wrong on this?
Is this a good way of doing? Does Google know nonetheless? For example from Chrome browser stats sent back. And does this maybe penalize ranking?
I’ve been meaning to respond to this for a while; it’s just I’m a master procrastinator.
At the end of the day, I’m keeping the default Discourse behavior because everything else is just too much to maintain.
I’m pretty confident that during my experiments with this, the JS version was better received by Google. The impressions seemed to always jump when it got the better version.
John Mueller of Google search responded to our tweets about this, here is the initial tweet.
I still think some sort of solution done by the open source community would be majorly beneficial. Perhaps server-side rendering.
I understand this would be a major thing for Discourse to take on, but imagine the jump in traffic to all the Discourse forums out there if Google got the nicer UI/layout, etc.
John of Google says it should be fine though, so that should be factored in.
I see, if I could add some input… from what I understand, the googlebot crawler is pretty good at finding new content/pages quickly regardless of stack behind the setup. That can’t be the only reason Discourse is using the crawler version but do understand the implications.
Maybe now with AI and how the internet is changing the idea of on-page SEO will become more and more of an irrelevant factor and the content/data itself is what will matter the most.
@stance455 If Discourse serves the JS version, Googlebot will notice how long it is until the (real) content shows up. Which is very bad for SEO.
The problem is only when Discourse is loaded initially. But that’s an important metric for people clicking on a link on a search engine (and thus, for Google themselves too).
