I am not sure why Google isn’t indexing the content of our discourse community. Look at this page here: Welcome to the Appsmith Community | Appsmith Community Portal The title is fairly unique. I’d imagine, when I copy and paste the title in Google, the first link should be the URL above. Instead what I get is this: Not only is it not the first link, it doesn’t show up at all. [Screenshot 2022-02-14 at 12.51.05 AM] What am I doing wrong?

here is an extract from Google The indexing of your content by Google is determined by system algorithms that take into account user demand and quality checks. [image] In-Depth Guide to How Google Search Works | Google Search Central | ... Get an in-depth understanding of how Google Search works and improve your site for Google's crawling, indexing, and ranking processes. Also remember that only TL3 and above users has the no follow links disabled. Here is something to read [image] Available site settings to configure "nofollow" for links posted by users Site Management bookmark This documentation explains how to configure “nofollow” attributes for links posted by users on Discourse forums. person_raising_hand Required user level: Administrator Default “nofollow” implementation By default, all user submitted links will have rel="noopener nofollow ugc" added to deter inappropriate or potentially damaging behaviour. The nofollow attribute tells search engines not to endorse the link, while ugc identifies it as user-generated content. This includes all li… In other words, my posts has more “weight” than your posts.

[image] JammyDodger: I believe that’s a TL3 perk by default. :+1: my apologies, you right :+1:

@constantine It looks like Google is prefering pages on the sub-domain https://docs.appsmith.com over pages on https://community.appsmith.com Most search results are from docs.appsmith.com Google Search A shot in the dark: Googles preference for one sub-domain over the orther might be influenced by “Core Web Vitals” Largest Contentful Paint (LCP) The simulated report on https://pagespeed.web.dev/ is “bugged” by Discourse. Better check the Google Search Console report “Experinece” → “Core Web Vitals” for your domain.

Just an update folks, we self-hosted discourse and submitted the sitemap to search console and now our content is being indexed by google. So perhaps something is up with the cloud hosted version?

I am not sure this is related to sitemaps or cloud hosting. Meta is hosted on AWS which is a completely different place to where we host many of our other customers and we started seeing very uneven results for meta lately and quite a few sites across various hosting options. I have been trying to tune a few things to see if anything helps. We no longer follow links to .rss which saves google from scanning /1 /2 etc variants of a topic that all share a canonical. We explicitly tell Google not to follow links inside the .rss feed in case it gets an rss feed. I temporarily disabled some canonical tuning we did - which showed promise: Search engines now blocked from indexing non-canonical pages The symptom I am observing here on meta is that Google is indeed crawling ALL the content, I can see that in the weblogs Despite crawling the pages on 50% or so of recent new meta topics are not showing up in the index. This is extremely concerning, Google is giving us very little visibility of “why?” here. My next step is to get more data and an on going report going we will probably use serpapi to figure out which pages are missing from Google and try to figure out a pattern.

[image] sam: Google is giving us very little visibility of “why?” here Because Meta, meaning OPs here, doesn’t use ”why” as Google wants? I did some expriences and even the context stays same, using different sentences changes search results. As expected. And… personal search history is big and not so good thing. My results are quite often pointing somewhere else than to Meta, because I’m doing searches here, not via Google. Yes. Still Discourse may have some issues with Google, and that may or may not come from ”forum status” where Google isn’t reacting to forums same way as to ordinary website. Or there is some technically issues — even that is the reason quite rarely.

[image] constantine: Just an update folks, we self-hosted discourse and submitted the sitemap to search console and now our content is being indexed by google. It’s the same thing I did, after an initial period of improvement then it returns to being ignored. After two months now even the new threads in the sitemap are not being indexed, it is a situation that has been going on for months and we have lost a lot of views. We want to try this for one month and if it doesn’t improve we will change software :sob:

[image] sam: figure out which pages are missing from Google and try to figure out a pattern. Check Google Search Console report “Index” → “Coverage” for the “missing” pages. This might not show you “why” but “what is going on”. Open “Index” → “Coverage” for your domain. Select "All submitted pages [grafik] See especially “Excluded” [grafik] Description of the different stages: Page indexing report - Search Console Help My assumption on Google: “Crawled - currently not indexed” has some upper limit set by Google - while there are too many pages in this state, new pages will only very slowly enter the Google index. On Google Crawler Performance see Google Search Console report “Settings” → “Crawl stats” Here especially the timeline and development of the “average response time” is interesting: faster response = more crawl requests [grafik] And also “By purpose” is interesting: [grafik]

My issue is, still, noindex-tag. Google is bypassing all aswers in topics because of noindex, but those things users are googling — and indexing covers just questions. Well, that can be fixed.

I also noticed that google, unlike bing, ignores the title of the discussion and I don’t understand why. This never happens with my other forum on SMF…

Why isn't Google Indexing Discourse? SEO concerns

Product Support

constantine March 14, 2022, 12:59pm 30

How does one remove the noindex tag?

Topic		Replies	Views
Google Search Indexing and Discourse Data & reporting	9	4030	June 9, 2020
SEO - Pages crawled but not Indexed Product Support seo	4	390	September 23, 2025
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Product Support	18	432	December 18, 2024
Sitelinks in Google disappearing Community Building	26	1878	January 27, 2023
1 million Topics - Takes millions of days to get indexed without Sitemap in Robots Product Support	3	2728	February 20, 2017

Why isn't Google Indexing Discourse? SEO concerns

Related topics