# Why isn't Google Indexing Discourse? SEO concerns
**URL:** https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098
**Category:** Support
**Tags:** seo
**Created:** [February 14, 2022, 8:56am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098 "2022-02-14T08:56:41Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@constantine](https://meta.discourse.org/u/constantine)
#### Post date: [February 14, 2022, 8:56am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/1 "2022-02-14T08:56:41Z")
I am not sure why Google isn’t indexing the content of our discourse community.
Look at this page here: [Welcome to the Appsmith Community | Appsmith Community Portal](https://community.appsmith.com/t/how-do-i-rewrite-my-entire-frontend-with-appsmith/555)
The title is fairly unique. I’d imagine, when I copy and paste the title in Google, the first link should be the URL above.
Instead what I get is this: Not only is it not the first link, it doesn’t show up at all.
What am I doing wrong?
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [February 14, 2022, 9:10am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/2 "2022-02-14T09:10:38Z")
There could be soooooo many reason for this.
Is the googlebot actually crawls your site ? check [mysite.com/admin/reports/web\_crawlers](http://mysite.com/admin/reports/web_crawlers)
Is the googlebot blocked or rate limited? check [mysite.com/admin/site\_settings/category/security?filter=crawler%20user%20agents](http://mysite.com/admin/site_settings/category/security?filter=crawler%20user%20agents)
Did you add your site to [Google Search Console](https://search.google.com/search-console/about) ?
Self-hosted people can even install the following plug-in to help
> [@Discourse Sitemap](https://meta.discourse.org/t/discourse-sitemap/40348):
>
> discourse2Summary Discourse Sitemap adds a sitemap.xml to your Discourse instanceopen_bookInstall Guide This plugin is bundled with Discourse core. There is no need to install the plugin separately. Features There has been [some](https://meta.discourse.org/t/how-to-create-xml-sitemap-is-there-any-plugin/33654) [discussion](https://meta.discourse.org/t/sitemap-xml-for-google-webmaster/9054/9) over the need of a sitemap, and I think Discourse doesn’t really need one, although I do believe that a forum might be indexed a bit faster with one. Anyway, we created this plugin because of a customer that needed a sitemap for Google N…
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [February 14, 2022, 11:25am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/3 "2022-02-14T11:25:41Z")
> [@IAmGav](#):
>
> Self-hosted people can even install the following plug-in to help
The Sitemap plugin is available on our Business and Enterprise plans as well.
> **[Plugin directory](https://www.discourse.org/plugins)**
>
> Extend the functionality of your Discourse community with our official plugins.
> [@constantine](#):
>
> What am I doing wrong?
Probably nothing. For some reason Google seems to give the “How do I do X” a higher priority than the actual topic with that exact title. Why? I’m not sure. It might just be an AI making that decision based on unknown patterns.
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [February 14, 2022, 11:30am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/4 "2022-02-14T11:30:35Z")
> [@gerhard](#):
>
> unknown patterns
Aka Google Patterns that no one knows 😉
---
### Author:  [@constantine](https://meta.discourse.org/u/constantine)
#### Post date: [February 15, 2022, 4:38am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/5 "2022-02-15T04:38:36Z")
Well that’s a big issue then for us. We’ll look into this, but it sorta defeats the whole purpose then, coz we can’t expect people to “go through” every topic in the community. In practice, most people will google for their issue (even if they’re a member of the community) to arrive at the answer.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [February 16, 2022, 3:43pm UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/6 "2022-02-16T15:43:10Z")
It’s really hard to tell what’s going on with Google in your case. I took the liberty of taking a closer look at your crawler stats and on first glance it looks like the Google crawler isn’t visiting your community very often. Did you already try to gain insights from the [Google Search Console](https://search.google.com/search-console/about)? Maybe there’s some information there that could help in getting to the bottom of this.
I’m seeing the same behavior for a small portion of recently created topics here on Meta. I’ll discuss this with our team in order to find out if there’s something we can do or if it’s simply Google being Google. I’ll keep you updated.
Also, did you see our two blog posts on SEO?
- [8 Tips to Improve Your Community’s SEO](https://blog.discourse.org/2021/10/forum-seo)
- [Handling SEO for Discourse Communities](https://blog.discourse.org/2021/11/discourse-forum-seo)
---
### Author:  [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [February 16, 2022, 3:57pm UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/7 "2022-02-16T15:57:49Z")
This google behavior started months ago and sadly is getting worse.
> [@Sudden drop in traffic](https://meta.discourse.org/t/sudden-drop-in-traffic/198350/15):
>
> The decline continues in my two discourse sites, all previously indexed pages are becoming “Crawled - currently not indexed” and disappearing from Google search. I can understand on one site but is it possible that it happens with in two completely different forums?
The sitemap in my case didn’t help …
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [February 16, 2022, 6:35pm UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/9 "2022-02-16T18:35:57Z")
> [@IAmGav](#):
>
> Did you add your site to [Google Search Console](https://search.google.com/search-console/about) ?
Yeah this is the first question I’d ask.
---
### Author:  [@constantine](https://meta.discourse.org/u/constantine)
#### Post date: [February 22, 2022, 9:55am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/10 "2022-02-22T09:55:51Z")
Well…our site is [appsmith.com](http://appsmith.com), so all subdomains and subfolders of [appsmith.com](http://appsmith.com) would ideally be indexed when we submit [appsmith.com](http://appsmith.com) to GSC right?
I have gone ahead and added [community.appsmith.com](http://community.appsmith.com) (our discourse forum) to GSC anyways today, but not sure if that’s going to change anything.
---
### Author:  [@arkklo](https://meta.discourse.org/u/arkklo)
#### Post date: [February 22, 2022, 9:58am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/11 "2022-02-22T09:58:49Z")
I would add the subdomain as a standalone property in your GSC and submit a dedicated sitemap for the subdomain.
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [February 22, 2022, 10:15am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/12 "2022-02-22T10:15:19Z")
@constantine
Your forum is being indexed since May 2021
---
### Author:  [@constantine](https://meta.discourse.org/u/constantine)
#### Post date: [February 22, 2022, 11:48am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/13 "2022-02-22T11:48:52Z")
Yep 🙂
This seems to be an issue then. Coz we definitely can’t continue if google doesn’t index individual posts coz it sorta defeats the entire aim of having the community (since most folks will discover content from google - including our current users, instead of going through the categories on discourse).
And this doesn’t seem to be an issue just on our discourse, but even this forum ([meta.discourse.org](http://meta.discourse.org)).
To showcase this, I googled the title of this post, and this is the result I got:
Basically for folks who end up using the discourse community as a knowledge base (which ofcourse many companies do), this becomes a big issue.
We’ve followed pretty much most of the guidelines given in the 2 blogs on SEO that were shared above, our content is rich, detailed, highly technical. Yet when people google for it, it doesn’t show up.
So genuine questions guys: Given our use case (which is creating a knowledgebase via support queries, which can also help in SEO), and seeing the issue at hand, would you recommend that we just start planning on finding an alternative?
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [February 22, 2022, 11:53am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/14 "2022-02-22T11:53:42Z")
here is an extract from Google
> The indexing of your content by Google is determined by **system algorithms** that take into account **user demand** and **quality checks**.
> **[In-Depth Guide to How Google Search Works | Google Search Central | ...](https://developers.google.com/search/docs/fundamentals/how-search-works)**
>
> Get an in-depth understanding of how Google Search works and improve your site for Google's crawling, indexing, and ranking processes.
Also remember that only TL3 and above users has the `no follow` links disabled.
Here is something to read
> [@Available site settings to configure "nofollow" for links posted by users](https://meta.discourse.org/t/nofollow-link-settings/93004):
>
> bookmark This documentation explains how to configure “nofollow” attributes for links posted by users on Discourse forums. person_raising_hand Required user level: Administrator Default “nofollow” implementation By default, all user submitted links will have rel="noopener nofollow ugc" added to deter inappropriate or potentially damaging behaviour. The nofollow attribute tells search engines not to endorse the link, while ugc identifies it as user-generated content. This includes all li…
In other words, my posts has more “weight” than your posts.
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [February 22, 2022, 12:22pm UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/16 "2022-02-22T12:22:16Z")
> [@JammyDodger](#):
>
> I believe that’s a TL3 perk by default. 👍
my apologies, you right 👍
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [February 22, 2022, 9:05pm UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/17 "2022-02-22T21:05:02Z")
@constantine It looks like Google is prefering pages on the sub-domain [https://docs.appsmith.com](https://docs.appsmith.com) over pages on [https://community.appsmith.com](https://community.appsmith.com)
Most search results are from [docs.appsmith.com](http://docs.appsmith.com)
[Google Search](https://www.google.com/search?q=site:appsmith.com)
A shot in the dark: Googles preference for one sub-domain over the orther might be influenced by “Core Web Vitals” [Largest Contentful Paint (LCP)](https://web.dev/lcp/)
The simulated report on [https://pagespeed.web.dev/](https://pagespeed.web.dev/) is “bugged” by Discourse. Better check the Google Search Console report “Experinece” → “Core Web Vitals” for your domain.
---
### Author:  [@constantine](https://meta.discourse.org/u/constantine)
#### Post date: [March 10, 2022, 6:58am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/18 "2022-03-10T06:58:28Z")
Just an update folks, we self-hosted discourse and submitted the sitemap to search console and now our content is being indexed by google. So perhaps something is up with the cloud hosted version?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [March 10, 2022, 7:54am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/19 "2022-03-10T07:54:50Z")
I am not sure this is related to sitemaps or cloud hosting. Meta is hosted on AWS which is a completely different place to where we host many of our other customers and we started seeing very uneven results for meta lately and quite a few sites across various hosting options.
I have been trying to tune a few things to see if anything helps.
- We no longer follow links to `.rss` which saves google from scanning `/1` `/2` etc variants of a topic that all share a canonical.
- We explicitly tell Google not to follow links inside the `.rss` feed in case it gets an rss feed.
- I temporarily disabled some canonical tuning we did - which showed promise: [Search engines now blocked from indexing non-canonical pages](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985)
The symptom I am observing here on meta is that
1. Google is indeed crawling **ALL** the content, I can see that in the weblogs
2. Despite crawling the pages on 50% or so of recent new meta topics are not showing up in the index.
This is **extremely** concerning, Google is giving us very little visibility of “why?” here.
My next step is to get more data and an on going report going we will probably use serpapi to figure out which pages are missing from Google and try to figure out a pattern.
---
### Author:  [@Jagster](https://meta.discourse.org/u/Jagster)
#### Post date: [March 10, 2022, 8:28am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/20 "2022-03-10T08:28:36Z")
> [@sam](#):
>
> Google is giving us very little visibility of “why?” here
Because Meta, meaning OPs here, doesn’t use ”why” as Google wants? I did some expriences and even the context stays same, using different sentences changes search results. As expected.
And… personal search history is big and not so good thing. My results are quite often pointing somewhere else than to Meta, because I’m doing searches here, not via Google.
Yes. Still Discourse may have some issues with Google, and that may or may not come from ”forum status” where Google isn’t reacting to forums same way as to ordinary website. Or there is some technically issues — even that is the reason quite rarely.
---
### Author:  [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [March 10, 2022, 8:30am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/21 "2022-03-10T08:30:46Z")
> [@constantine](#):
>
> Just an update folks, we self-hosted discourse and submitted the sitemap to search console and now our content is being indexed by google.
It’s the same thing I did, after an initial period of improvement then it returns to being ignored.
After two months now even the new threads in the sitemap are not being indexed, it is a situation that has been going on for months and we have lost a lot of views.
We want to try [this](https://meta.discourse.org/t/search-engines-now-blocked-from-indexing-non-canonical-pages/218985) for one month and if it doesn’t improve we will change software 😭
---
### Author:  [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [March 10, 2022, 11:45am UTC](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098/22 "2022-03-10T11:45:17Z")
> [@sam](#):
>
> figure out which pages are missing from Google and try to figure out a pattern.
Check Google Search Console report “Index” → “Coverage” for the “missing” pages. This might not show you “why” but “what is going on”.
1. Open [“Index” → “Coverage”](https://search.google.com/search-console/index?hl=en) for your domain.
2. Select "All submitted pages
3. See especially “Excluded”
Description of the different stages: [Page indexing report - Search Console Help](https://support.google.com/webmasters/answer/7440203#crawled)
> [@](#):
>
> **[Crawled](https://support.google.com/webmasters/answer/7643010) - currently not indexed:** The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.
>
> **Discovered - currently not indexed:** The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.
My assumption on Google: “Crawled - currently not indexed” has some upper limit set by Google - while there are too many pages in this state, new pages will only very slowly enter the Google index.
* * *
On Google Crawler Performance see Google Search Console report [“Settings” → “Crawl stats”](https://search.google.com/search-console/settings/crawl-stats?hl=en)
Here especially the timeline and development of the “average response time” is interesting: faster response = more crawl requests
And also “By purpose” is interesting:
[Next page](https://meta.discourse.org/t/why-isnt-google-indexing-discourse-seo-concerns/218098.md?page=2)