SEO - Pages crawled but not Indexed

Support
1

Hello,

I am a community maintainer for a prominent open-source project’s, Keycloak, Discourse group. We noticed starting this last January that the Discourse site stopped showing up in Google search results. They still appear in Bing and other search engines.

Does anyone have good insight or experience about why Google might stop seeing the content as valuable or crawl but not Index the content? The error from the Google webmaster tools is “Crawled - currently not indexed”. Testing live urls point that all is ok.

We already reached out to Discourse support and they are mystified as well. They suggested a community post for any experts that might be on the forums and could assist.

Site: https://keycloak.discourse.group/

Sitemaps are setup correctly.

2

Some additional items (limited by Discourse’s new member limits)

image
image1774×1050 72.4 KB

3

Sitemap

image
image1934×1000 108 KB

4

We have seen this problematic pattern in the past with some other sites.

In many ways, Google is a black box and it behaves in mysterious ways. That said, there is some general guidance that I think is important to follow:

Establish authority for your domain

Google is built on PageRank. The more inbound links your forum has, the better the PageRank.

When I search for keycloak.discourse.group, the top link is on meta (this topic), but number #2 is:

image
image1083×956 135 KB

The #2 authoritative answer Google has is “don’t use keycloak.discourse.group”

Additionally, there is no “authority” of where discussion should live:

  • GitHub is fine
  • Slack is fine
  • Mailing is fine
  • Discourse is fine

Instead, centralize on one platform for async discussion. You now have 3 options for a pretty small (in the big scheme) amount of discussion.

Mailing list is getting 20-30 posts a month.
Discourse is getting a bit more
GitHub is getting similar traffic to Discourse

Pick a winner and run with it; this is how you establish authority

Bias toward capitalizing on your existing authority

When it comes to choosing a domain, you already have authority on your existing https://www.keycloak.org/ brand.

When designing a community, your preference should be to grow your online brand.

community.keycloak.org or forum.keycloak.org is what you want. Not Reddit, GitHub, or even discourse.group.

Additionally, have a footer and link to this forum / async community portal from every page.

4 Likes