Why is Google not crawling my site?

[dunebuggie (Shreyas)]

Hey all,

Previously we used to get a good number of discussions from our community forum when we search for some topics on Google. But recently, we noticed that we don’t see any results from the community in the search results. We’ve done some re-categorization and closed a bunch of discussions recently. That was the only change.

We also tried changing the robots.txt to allow everything: https://community.tyk.io/robots.txt

However, we’re still getting a 403 Forbidden on Browseo: https://www.browseo.net/?url=https%3A%2F%2Fcommunity.tyk.io%2F

As of now, when we search for the community on Google, this is the result:

Unsure what we’re doing wrong. Any help would be much appreciated. Also, in general, if there are any blogs or tutorials around SEO on Discourse communities, please do share. Would love to know how we can improve.

Thanks!

[simon (Simon Cossar)]

What do you see if you click on the “Learn why” link that’s displayed at the bottom of the Google search result?

Have you made any changes to your Discourse site’s whitelisted crawler user agents or blacklisted crawler user agents site settings?

[codinghorror (Jeff Atwood)]

You don’t have to do anything in our years of experience. However, if your site is not well linked on the web it may take a while. See the Google Webmaster Tools for your domain.

[dunebuggie (Shreyas)]

@simon

What do you see if you click on the “Learn why” link that’s displayed at the bottom of the Google search result?

I’m taken to this page on Google Support.

@simon

Have you made any changes to your Discourse site’s whitelisted crawler user agents or blacklisted crawler user agents site settings?

Don’t think so.

[dunebuggie (Shreyas)]

Thanks, Jeff. I’m getting the team to look into Google Webmaster Tools. Most likely a Google issue because we really haven’t changed anything on Discourse, other than closing threads and recategorization.

[Canapin (Coin-coin le Canapin)]

I don’t know about sitemaps but I’ll quote Discourse’s team on this:

@justin

the sitemap plugin isn’t necessary for sites to be indexed by Google as Discourse is already structured to allow crawlers to index it easily. But for large sites, the plugin can make it marginally faster.

@codinghorror

Site maps are useless and unneeded.

[riking (Kane York)]

@dunebuggie

@simon

Have you made any changes to your Discourse site’s whitelisted crawler user agents or blacklisted crawler user agents site settings?

Don’t think so.

whitelisted crawler user agents
WARNING! SETTING THIS WILL DISALLOW ALL CRAWLERS NOT LISTED HERE!

The name being in yellow means the setting was modified.
Clear that setting with the reset button.