If you have a chance, it would be great if you could test a few of the URLs that are being marked as “Crawled - currently not indexed.” This can be done with the Live URL test. Details about running the test are here: URL Inspection Tool - Search Console Help.
3 Likes
I’m trying to improve Value to Excluded ratio. One possible improvement, I believe, is the sitemap. Can I edit the sitemap on Discourse? Where is it and how do I get access?
1 Like
By default, Discourse doesn’t include a sitemap, but you can add one with this plugin: Discourse Sitemap
2 Likes
Do you have any more information on a possible change to the timeout? I looked at the site content using Google Search Console and the top 15 responses are for terms that aren’t a good match for the content. It’s likely that the person clicking on the search term link on the Google page didn’t find what they wanted in the discussion on the Discourse forum we are analyzing.
It’s also not clear why Google would direct them to the article in the first place as the article doesn’t appear to contain some critical keywords. For example, although it does “camera” for a 360 consumer camera in the article, the keywords used to find the article contain CCTV, which has nothing to do with a handheld consumer 360 camera.
Update: August 22, 2021
I don’t know how to interpret or use Google Search Console properly. Is there any idea what the 679 failing means?
The report above is from this area:
2 Likes
That’s right, all the discussions in my case were reworked by google and disappeared from the search.
I have no idea, for now on my sites I still have:
Not enough usage data in the last 90 days for this device type.
The decline continues despite there is always new content and updated discussions.
No errors are reported, it is clear that something has happened!
2 Likes
Are you on the latest version of Discourse?
3 Likes
We’ve got 2.8.0.beta11 installed now, it looks like we need to install 2.9.0.beta4.
3 Likes
.beta6 now… reminds me I gotta upgrade! 
2 Likes
Never had sitemap using plugin, now that it’s native and plus various fixes on latest version for about a month noticeable change in indexing, some was close to instant, not sure it’s helping traffic but some indexing activity has to be welcome.
2 Likes
Since we did the upgrade from very early 2.x version to latest 2.9, this is how the coverage looks now, it took about 12 days for that uptick to start. “2” denotes approx when the latest version was achieved.
The historical trend was a declining indexing of valid links, but you can see the trend is reversed, i.e. it has had an effect of increasing indexing activity overall, at least.
There is approx. a net increase of 15K valid links.
While it is not totally clear in the above graphic, the invalid links went up a little post upgrade, but have since started to trend back down.
Slowly, and the pages invalid with a redirect, were the main and largest invalid group and it is now showing a slow but downward trend for the last month.
Less crawling and more is now directly sourced via the sitemap, a very clear before vs after sitemap effect:
One more before and after, breadcrumbs (and sitelinks had similar take up)
Tbh I will give it more time, at least 3 months to see how it all pans out because it was a big jump from version and going from no sitemap after a couple foyers to a site map - in the meantime I will try keep pace with the upgrade.
Some obvious improvement here too, things went weird for a bit current state looks improved overall for more vitals:
The only technical improvements that might be helping speed (on top of any programatic improvement with the PG13 DB and Discourse codebase) is that the HD would be nVME, and a very updated docker too - definitely noticing a big improvement in performance overall and speed tests look very good now using google pagespeed.
Overall the general concerns on negative trends in google console appear to be turning around with version 2.9.beta4-6, albeit slowly.
4 Likes
Also note that we recently added
which will help with this 
6 Likes
It’s likely that the person clicking on the search term link on the Google page didn’t find what they wanted in the discussion on the Discourse forum we are analyzing 
I dunno what changed. Maybe update, but around start of October, suddenly Google is now indexing GOOD Urls for desktop.
Discovery suddenly began October 1-3 and the sudden uptick around 25th October in the curve coincides with the flip above, I dunno is this google seeing things properly or an update improving indexability.
It’s still not indexing for mobile.
This deployment is only a few commits away from totally up to date - 2.9.0.beta10 [f7a4fd1f49]
1 Like
Yes it is. Poor URLs is totally different thing than not indexing.
1 Like
This happened: Introducing Discourse Splash - A visual preloader displayed while site assets load
That’s a good point. @agemo is using “indexing” and Google’s scoring of “good/medium/bad” Web Vitals interchangeably, when they are absolutely not the same thing, even though the later may affect ranking.
2 Likes
Well it’s funny you should pedantically point that out, and it’s a technical point, but it’s missing the substantive overall point, cause sometimes language - this actually coincides with an uptick in indexing activity, that I didn’t post. So I’m waxing on the derivative, as it’s all derivative of the indexing service…
1 Like
Well see I don’t get that because that was around July and so it’s 3 months later before any GOOD urls were being seen by google as per above screenshots, I would have hoped sooner on this deployment as it has had the splash screen commit within days of it’s initial release.
I suspect when I removed some very simple but old custom mod for a banner (CSS/HTML) and reverted to using versatile banner component instead, then things clicked into place correctly, though it would not explain the GOOD urls appearing start of october, but it might explain the october surge.
I was aiming to reduce CLS issues, and so I am guessing the pages render and appear better. Maybe I’ve solved a deeper issue here. IIRC versatile banner is not being used on mobile views.
1 Like
Another thing that has also happened, and is why I started to look again closely in the first instance, is that non-indexed pages discovery has started to increase with a drop in index too, hmmm…all around the last week or so in October, when all these things seem to kick in at once.
I’ve looked at the non-index / 404’s, they seem to be a main source of the recent non-index rise, and are now close to 10,000, they appear as below for e.g.:
site.com/topic-name/47168?page=137
site.com/topic-name/470?page=491
If I remove the ?page=123 the url resolves correctly, so the page is there - where is google fishing this incorrect link formatting? 
I was thinking it was an error from phpBB import or some other permalink artefact/setup-incorrectly, or something else.
2 Likes