Thanks, Josh, appreciate the feedback here.
Nice catch and while poorly executed, taking many months , - Google seems to have figured out what pages mean what.
In other words, eventually, I started to see more and more of the pages that I 301’ed to for the keywords used on the old pages
This makes a lot of sense and will see how I can get that impelelnted- presently the search console doesnt show the crawler getting 301 to often. Seems when the rank gets better, they will follow more 301. Causation without correlation perhaps.
It’s totally not a knock on Discourse - I’m just not easily convinced with " thousands of Discourse users have great organic traffic"
Google is not really going to tell us either.
We must always remember Google is an algorithm, they’re not looking at this from a human’s eyes.
While both versions share similar content, and google knows it’s not malicious cloaking- they still have to adjust rank.
One version looks way better, works better, and gives some sense of internal link structure. The other is a glorified RSS feed.
Google has no idea I have this slick forum that works on all [modern] devices, truly encourages discourse, and is one of the coolest things the internet has ever created.
I always like to use the “Powered by discourse” do-follow link in the crawler version. ( just because its easy )
Again, I know not malicious but you must look at it through Google’s eyes. You FlyNumber ( not https://community.cloudflare.com/ ) are giving us this crawler version with an external link you are not showing regular browsers.
I could totally see the algorithm picking up on whats going on and ignoring the external link for the cloudflare domain (as it’s such an authority)
It’s not like what google applies to cloudflare will apply to me.
Did someone pay you for this external link you show bots ( but don’t want to show regular users? ) is more about how they may look at the site. I’m not saying it’s this - but it’s a possibility you’ll want to eliminate.
In simplest terms, the crawler version doesn’t have a menu or any real structure.
That’s the content the algorithm thinks you want to serve to end-users.
From a very general perspective, I can’t see the algorithm rewarding that.
Maybe it’s time we start considering a real overhaul of the crawler version. At least add the main menu, and suggested topics on the bottom.
Interesting update Google has added “JSON” to the “file types” in crawl stats for my discourse instance. “Javascript” is a separate “file type”.
Will follow closely but would still love for this to render properly in the google tool.