What conditions render a post unable to be crawled by search engines?

support
1

What conditions render a post unable to be crawled by search engines?

In the life of a Topic, under what conditions does it become crawlable/indexable for the search engine crawlers?

Some possibilities (and I’d suppose others not mentioned here) that I wonder about are:

  • when an original Topic is drafted, submitted, and gets caught in a Watched Word filter (Blocked or Require Approval)

  • when a Topic gets Unlisted, is it then removed from ‘crawlable’ status

  • when a Topic gets Deleted, is it then removed from ‘crawlable’ status

  • when a post gets Flagged, is it then removed from ‘crawlable’ status

I suppose the question might be answered in terms of what conditions make a post NOT crawlable.

1 Like
2

Yes, all of the above is correct. The other thing that can prevent a topic from being crawled is to put the topic in a category that is configured to not allow anonymous users to access it. All topics on a Discourse site can be made uncrawlable by enabling the login required site setting.

You can also prevent your site from being crawled by disabling the allow index in robots txt site setting. That setting is enabled by default.

4 Likes