Glad you asked, yes, I think it’s the reason for my google “soft penalty”
Allow me to elaborate.
We had a very sloppy site update around sept/oct 2019 , the main site tanked right then and there.
We never recovered. The site has never been better as far as SEO. Sure its not perfect but we’re light years ahead of some of the competition. Sites that use our many years’ old images and text outrank us by pages. We’re on the 3rd and they perhaps are top of 2nd page.
I’ve been through countless SEO blogs, videos, posts and even had some back and forth with John Mueller (on Reddit )
The most I got out of him was, it could be “quality issues”. We had improved the main site dramatically since Jan 1 of this year. Not even a blip in organic traffic.
Discourse: I had it installed back in 2013 and forgot about it. Barely would check its traffic.
If you look at the main site analytics, you’ll see a sharp drop towards the end of the chart. This is when I started working on Discourse.
When trying prerender.io on discourse the rank for the main site was all over the place. Sometimes jumping 10-15 spots overnight, then back. (I have since stopped prerender as they couldn’t render the main menu , login etc. )
From what I read online, this is a sign google doesn’t know where to place you. They say just a little “more” and you’re on the good side of the algorithm.
Nothing we’ve done in the last 3 years has triggered these fluctuations in the SERPs.
(Messing with Google disavow tool, cleaning up code, clean URLs, site structure, internal linking, social, content, etc.)
You might make the argument, why didn’t google penalize you in 2018? ( you had discourse on the subdomain then too)
Well, I think it was a multitude of factors unique to the site, it’s history, link profile. that caused it to tank in late 2019. Seems that google reshuffled the site rank and perhaps gave the discourse URLS more weight than what it gave previously.
And the thing is … I love Discourse. Especially now that I’ve been on meta more, all these cool plugins and features I had no idea existed. Wiki, subscription payments, table of contents, and now the chat !!
So moving away from discourse is not really an option, too much invested at this point.
I did consider this and I’m willing to take my chances. I know it won’t be perfect but from what I read and watch , Google has gotten really good at understanding JS as of late.
They even deprecated the ajax crawling scheme
Times have changed. Today, as long as you’re not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.
Side note: Discourse has a setting for ajax crawling- I guess that has to eventually go
So the plan is to serve the app to Google, do my best to fix any SEO issues that may arise and enjoy the spike in traffic.
I can then report the results on meta and make the case that Discourse should consider optimizing the JS for Google.
For example, maybe something like this (from google blog) would help with the pagination and scrolling concerns.
And keep the non-crawler version for old browsers.