Googlebot is getting non-javascript version of the site

So I just noticed that Discourse serves a non-JS version of the site to Google…

I’ve been using JS to set the sites title, meta description, canonical, etc and seems google is getting really good at rendering client-side javascript ( I’m referring to my regular site, not discourse )

Is there a way to disable this and not give google bot a specific version?

…Also, discourse has a do-follow link there for itself “Powered by Discourse.” - Is there anything I can do about that?

Edit 1 : I should also add, I don’t mind having a Discourse link in there, perhaps no-follow , or do-follow but not site-wide

Edit 2:: Im on github and see I’ve got options as far as removing the link- are there any suggestions as to which I should be leaning towards?

1 Like

Anyone have any insight here?

Probably a good idea to explain why this is necessary. The out-of-box behavior within Discourse works fine for the majority, based on your account it’s your choice with the above which is creating the conflict.

3 Likes

My apologies

When I say I’m using JavaScript for the title, canonical and meta-description I mean for my regular website, not the discourse setup. I was just trying to convey my experiences with googlebot and JS

I tried adjusting the robots file, that didn’t seem to serve the JavaScript version to google

Edit: Im also starting to grasp how much of discourse is an app , client-side - very cool stuff . Still would be interesting to see what the google bot can do with it. I imagine the algorithm would really like the clean setup.

1 Like

Right. It’s all an app. And there is little chance that Google could make sense of it, so there would need to be a second app, just for Google, which is why it sends the static version.

1 Like

Thanks for replying here @pfaffman

Just to scratch the itch, is there a way to test this? - Im just curious what the site will look like if we give google a try. They’ve started to really open up to the idea of client-side JS as of late. SEO wise at least it makes sense to try

The non-js version serves hyperlinks in the same color as the text. So you cant tell what a link is ( if you’re google bot )

The source code is available, so you’d just change something there. I am pretty familiar with the code, but have no idea where to look.

I think that if that were true there would already be an answer, as a bunch of people who know more about Discourse than I do and more about SEO than either of us care pretty deeply about this matter.

Do you really think that the Google bot is color blind and looks at text color to decide what is a link? And it isn’t able to read the HTML to see what is an href? That seems improbable.

It is actually totally color blind :wink:

Quite many, even SEO experts, really thinks bots see a site same way as we see it on screen.

Well, I’m kind of out of topic now.

3 Likes

I was thinking of starting a new topic here on meta to discuss this (Im not familiar enough with this pull requests/ commit jargon on github)

Trust me, Im shocked too… I also wouldn’t call myself a beginner when it comes to SEO, I’ve been on and off being involved for well over a decade. Whats going on here are negative ranking factors for sure. It may not hurt “LetsEncypts” SEO but smaller guys, for sure.

I think the majority of discourse users may share your mentality as far as SEO. Not giving it much thought. There’s a ton of hidden potential here.

Yes they do know its a link, they also know its dark just like the text and users won’t like that. Let’s not forget, google is now ranking us on what they see in the non-JS version.

In regards to the “Powered by Discourse” external link:

I remember watching a video with John Mueller/Matt Cutts - the Google SEO guys and they mentioned hidden internal links ( due to menu’s, JS etc ) dont really do damage. -

He also mentioned though, if its an external link , that you only show googlebot and not end-users, that it could be an issue.

I feel like im in the twilight zone to be honest. I guarantee you, once this is fixed , all Discourse forums will see a huge jump in Google organic traffic

It’s really easy to make claims like the above, particularly when by your own admission you don’t understand the inner workings nor design decisions which led to the current state.

Realistically speaking though you can’t guarantee anything here, and unless you can convince someone with the requisite skills to alter the code it’s going to remain hypothetical.

If you have a budget to explore this the #marketplace might be a good starting point.

3 Likes

I have a very broad idea, the app is Javascipt heavy and historically Google has had issues with JS. So you guys went with the simple non-JS we have now. My argument is to at least TRY and see what google renders with JS enabled.

I actually think I’m underselling this, some agree with me over on Reddit technical SEO

Side note:
It seems the hyperlinks in the body being the same color as the text was in a somewhat recent update. Older versions of my page on google show the links are blue.