# How public Discourse sites are indexed by search engines like Google

**URL:** https://meta.discourse.org/t/how-public-discourse-sites-are-indexed-by-search-engines-like-google/1184
**Category:** Site Management
**Tags:** reference
**Created:** [6 בפברואר,‏ 2013,‏ 12:14pm UTC](https://meta.discourse.org/t/how-public-discourse-sites-are-indexed-by-search-engines-like-google/1184 "2013-02-06T12:14:19Z")
**Posts on this page:** 1
**Showing post:** 1

<div class="post-metadata">

### Author: ![Discourse](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/discourse/32/148734_2.png) [@Discourse](https://meta.discourse.org/u/Discourse)
#### Post date: [6 בפברואר,‏ 2013,‏ 12:14pm UTC](https://meta.discourse.org/t/how-public-discourse-sites-are-indexed-by-search-engines-like-google/1184/1 "2013-02-06T12:14:19Z")

</div>

> 🔖 This reference explains how public Discourse sites work with search engines like Google and how the platform ensures content is properly indexed even as a JavaScript application.
> 
> 🙋 Required user level: All users

# Search engine indexing of Discourse sites

Discourse is built as a JavaScript application, but is specifically designed to ensure search engines can properly crawl and index all content on public sites.

## How Discourse supports search engine crawlers

While Discourse uses modern JavaScript for its interactive features, it implements several techniques to ensure search engines can properly index all content:

### Dedicated crawler layout

Discourse automatically detects search engine bots by their user agent using the `CrawlerDetection` module. When a crawler is detected, Discourse serves a **completely separate server-rendered HTML layout** (`crawler.html.erb`) instead of the normal JavaScript application. This crawler layout includes:

1. Full HTML-rendered topic content and topic lists — no JavaScript required
2. [Schema.org](https://schema.org) structured data markup (e.g., `DiscussionForumPosting`, `ItemList`, `BreadcrumbList`) to help search engines understand your content
3. Proper pagination with `rel="prev"` and `rel="next"` links to allow complete crawling
4. `Last-Modified` headers on topic pages to signal content freshness

### No-JavaScript fallback

For regular browsers that have JavaScript disabled, Discourse also includes a `<noscript>` tag in the standard application layout. This contains rendered topic lists and topic content, ensuring the site remains accessible even without JavaScript.

## Robots.txt and indexing controls

Discourse provides several settings to control how search engines interact with your site:

### `allow_index_in_robots_txt`

This site setting (enabled by default) controls whether your site’s `robots.txt` permits crawling. When disabled, the `robots.txt` will disallow all crawlers and an `X-Robots-Tag: noindex` header is added to all responses.

### Crawler allowlist and blocklist

You can control which crawlers are allowed to access your site using:

- **`allowed_crawler_user_agents`** — when set, only the listed crawlers are permitted; all others are blocked via `robots.txt`
- **`blocked_crawler_user_agents`** — when set, the listed crawlers are blocked while all others are allowed

### Custom robots.txt

Admins can fully customize the `robots.txt` file at `/admin/customize/robots`. A customized robots.txt overrides the default generated one entirely.

### Default disallowed paths

By default, Discourse blocks crawlers from paths that aren’t useful for indexing, such as `/admin/`, `/auth/`, `/email/`, `/session`, `/search`, and others. Googlebot gets a more permissive configuration, only blocking the core admin/auth paths.

## Viewing the crawler version

You can view how search engines see your Discourse site by:

- Installing a JavaScript-disabling browser plugin in Chrome or Firefox
- Using Chrome DevTools to disable JavaScript ([Google’s instructions](https://developer.chrome.com/docs/devtools/javascript/disable))
- Changing your browser’s user agent string to a known crawler (e.g., `Googlebot`) to see the dedicated crawler layout

> Last edited by @jessii 2025-05-21T22:32:48Z
> 
> > **Check document**
> >
> > Perform check on document:

---

_[View the full topic](https://meta.discourse.org/t/how-public-discourse-sites-are-indexed-by-search-engines-like-google/1184)._