Why is Google not crawling my site?

Hey all,

Previously we used to get a good number of discussions from our community forum when we search for some topics on Google. But recently, we noticed that we don’t see any results from the community in the search results. We’ve done some re-categorization and closed a bunch of discussions recently. That was the only change.

We also tried changing the robots.txt to allow everything: https://community.tyk.io/robots.txt

However, we’re still getting a 403 Forbidden on Browseo: https://www.browseo.net/?url=https%3A%2F%2Fcommunity.tyk.io%2F

As of now, when we search for the community on Google, this is the result:
image

Unsure what we’re doing wrong. Any help would be much appreciated. Also, in general, if there are any blogs or tutorials around SEO on Discourse communities, please do share. Would love to know how we can improve.

Thanks!

What do you see if you click on the “Learn why” link that’s displayed at the bottom of the Google search result?

Have you made any changes to your Discourse site’s whitelisted crawler user agents or blacklisted crawler user agents site settings?

3 Likes

You don’t have to do anything in our years of experience. However, if your site is not well linked on the web it may take a while. See the Google Webmaster Tools for your domain.

2 Likes

I’m taken to this page on Google Support.

Don’t think so.

Thanks, Jeff. I’m getting the team to look into Google Webmaster Tools. Most likely a Google issue because we really haven’t changed anything on Discourse, other than closing threads and recategorization.

I don’t know about sitemaps but I’ll quote Discourse’s team on this:

1 Like

whitelisted crawler user agents
WARNING! SETTING THIS WILL DISALLOW ALL CRAWLERS NOT LISTED HERE!

The name being in yellow means the setting was modified.
Clear that setting with the reset button.

6 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.