SEO for discourse not working properly


(Johnny) #1

Hi,

My search engines are not updating my community e.g its been a week I posted new topics, which are no being shown on google yet. In the webmaster, robots.txt and sitemap is working fine for the domain. However, its not indexing.

in settings for the field exclude rel nofollow domains I have only given my domain.org. However, my community is based on a subdomain.

My /robotx.txt file has

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
User-agent: *
Disallow: /auth/cas
Disallow: /auth/facebook/callback
Disallow: /auth/twitter/callback
Disallow: /auth/google/callback
Disallow: /auth/yahoo/callback
Disallow: /auth/github/callback
Disallow: /auth/cas/callback
Disallow: /assets/browser-update*.js
Disallow: /users/
Disallow: /badges/
Disallow: /search
Disallow: /search/
Disallow: /tags
Disallow: /tags/

Sitemap: https://meta.domain.org/sitemap.xml

Any solution? How can I make SEO better for my discourse?


(Ed Arcos) #2

If Google is not indexing, or indexing is getting slower, it is about Google, not Discourse.

Also, there are hundreds of possibilities why this is happening, but I don’t think one of them has to do with the Discourse installation.


(Neil Lalonde) #3

As @frostydog says, it’s complicated…

The first step is to verify that crawlers can see the content. Use the chrome developer console to change your user agent to “Googlebot” and reload the page. If you see no topics or an error page, then something is broken in your install. I think Google’s web master tools will report problems too.


(Johnny) #4

@neil Hi, I have checked this. While loading the forum with google bot, its something as shown in the image below.


However, Now 2 days ago I posted a topic, on google, when I search about it, I don’t find the link to that topic, however I get that name in the categories as shown in the image below.


Same for most of the topics, the categories is getting published but the title to the topic is not being published. I have submitted by robots.txt as well as sitemap to Google and everything is fine. I also added the tag but that did not help as well. I added this tag as I found on several seo blogs that it helps. But in my case it did not. Also my server IP is not blacklisted.


(Evgeny) #5

Robots.txt - For Yandex, but these are the general rules

  1. specify the HOST
Disallow: /*.rss
Disallow: /latest?exclude_*

A lot of duplicate pages to see the search

https://meta.discourse.org/
https://meta.discourse.org/latest
https://meta.discourse.org/latest?exclude_category_ids%5B%5D=1
https://meta.discourse.org/latest?exclude_category_ids%5B%5D=2 … etc.

This is the data of my site (Search engine index - Yandex)


Negative SEO because of too much category pages getting indexed
(Neil Lalonde) #7

@john3 Not sure why google isn’t showing it in its index. Give it a few more days?