# Enabling native LLMs.txt support in Discourse
**URL:** https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088
**Category:** Site Management
**Created:** [September 25, 2025, 12:25pm UTC](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088 "2025-09-25T12:25:43Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Discourse](https://meta.discourse.org/u/Discourse)
#### Post date: [September 25, 2025, 12:25pm UTC](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088/1 "2025-09-25T12:25:43Z")
> :bookmark: This guide explains how to make your community discoverable and usable by AI agents and LLM crawlers through the `llms.txt` standard. It covers both the automatically generated default document and custom uploaded files.
>
> :person_raising_hand: Required user level: Administrator
### Summary
Discourse includes native support for the [`llms.txt` standard](https://llmstxt.org/), a proposed convention for providing an LLM-friendly overview of a website. Think of it as “robots.txt for AI”: a small Markdown file at `/llms.txt` that helps large language models (LLMs) and AI agents understand what your site is about and how to access it responsibly.
You can choose between two approaches, or combine them:
1. **Let Discourse generate a default document** automatically from your site settings
2. **Upload your own custom `llms.txt` file** for full control over the content
A custom upload always takes priority over the generated document.
### What is llms.txt?
`llms.txt` is a proposed standard that provides a curated, agent-friendly map of a website. It is served at the site root (`/llms.txt`) and follows a precise Markdown format:
- An `H1` heading with the name of the site (the only required section)
- A blockquote with a short summary
- Optional markdown sections with more detail
- `H2`-delimited sections containing lists of useful links
Unlike `sitemap.xml`, which lists pages for search engines, `llms.txt` is a concise overview designed to fit in an LLM context window. The detail lives behind the links, which are fetched only when needed.
Starting with [Discourse v2026.1.0](https://github.com/discourse/discourse/releases/tag/v2026.1.0), Discourse can serve `llms.txt` from the site root.
### How Discourse serves /llms.txt
Discourse handles requests to `/llms.txt` in this order:
1. If you have **uploaded a custom file** , Discourse serves it unchanged as `text/plain`
2. If no custom file is uploaded and the **generated default is enabled** , Discourse generates and serves the document
3. Otherwise, Discourse returns a `404`
The generated document is identical for anonymous visitors and signed-in users, and it is not affected by `login required` redirects.
### The generated default document
When no custom file is uploaded and the feature is enabled, Discourse builds a concise document using your site’s existing settings and the site’s default locale. It includes:
- The **site title** (from the `title` setting), with the hostname as fallback
- A **summary blockquote** using your `site_description`, or `short_site_description` as a fallback
- A **human community policy** stating that the site is for human discussion and that agents must only write when a human explicitly asks them to
- An **agent access policy** asking agents to fetch only what they need, respect `robots.txt` and crawler controls, and honor `HTTP 429` with the `Retry-After` interval
- A **Discourse MCP reference** , pointing agents to the [Discourse MCP server](https://github.com/discourse/discourse-mcp) as the preferred, permission-aware interface
- A **Preferred agent interface** section with the Discourse MCP setup link
- A **Public web access** section with links to `/search`, `/filter`, `/latest`, and `/categories`, plus `/sitemap.xml` when the `enable_sitemap` setting is on
- An **Optional** section with links to `/about`, `/guidelines`, `/tos`, and `/privacy`
If your site has `login_required` enabled, the document adds a note that content is available only to authenticated members and omits the public discovery links described above. Links are base-path aware, so they work correctly on subfolder installs.
You can see the announcement and discuss the generated template in [Automatically generated llms.txt](https://meta.discourse.org/t/automatically-generated-llms-txt/411435). Meta itself currently serves a [stock file](https://meta.discourse.org/llms.txt).
### Enable the generated default document
The generated default is delivered through the [Upcoming Changes](https://meta.discourse.org/t/upcoming-changes/395390) system as a beta change that admins can opt into or out of.
1. Go to **Admin → Configure → Upcoming Changes** , or visit `/admin/config/upcoming-changes`
2. Find the change that generates a default `/llms.txt` when no custom file is uploaded
3. In the **Enabled for** dropdown, choose **Everyone** to enable it for all visitors
4. To turn it off, choose **No one**
Promoted changes appear here automatically when they are ready, and admins are notified on the dashboard. If you prefer that no `llms.txt` file is served at all, set this change to **No one** or upload a minimal custom file.
### Upload a custom llms.txt
If you prefer to control the content exactly, you can upload your own file.
1. Prepare a `llms.txt` file in `.txt` or `.md` format (maximum file size is 512 KB)
2. Go to **Admin → Settings → Security** , or visit `/admin/config/security`, and search for **LLMs TXT**
3. Upload your file and save the configuration
Once configured, Discourse serves your file unchanged at:
```plaintext
https://yourforum.com/llms.txt
```
Refer to [llmstxt.org](https://llmstxt.org/) for syntax and format guidance. A custom upload always overrides the generated document.
### Verify your setup
You can check what agents will see by requesting the file in a browser or with `curl`:
```bash
curl https://yourforum.com/llms.txt
```
### **Best practices**
- Keep the file **concise**. `llms.txt` is a map, not a manifest; every token costs context, and a huge file leads to confused agents and worse results
- Only serve the generated default if its content matches your policies. If you need different wording, copy the generated text, edit it to suit your community, and upload it as a custom file
- Structure custom files per the spec: `H1` title, blockquote summary, then `H2` sections with link lists, using an `Optional` section for secondary links
- Combine `llms.txt` with crawler controls. To tune automated access, configure the `slow_down_crawler_user_agents` and `slow_down_crawler_rate` settings (Admin → Settings → Security)
- Remember that `llms.txt` describes your site, while `robots.txt` controls access. They serve different purposes and complement each other
### **Common issues and solutions**
**I get a 404 at `/llms.txt`**
This happens when no custom file is uploaded and the generated default is not enabled. Upload a custom file, enable the generated default in **Admin → Upcoming Changes** , or both.
**I use a llms.txt Generator plugin and the main file stopped working**
Core Discourse serves the `/llms.txt` path, so it takes precedence over plugins that generate it. If you rely on the plugin’s dynamically generated files, a practical workaround is to upload a minimal custom `llms.txt` that points agents to the plugin-generated content, for example:
```plaintext
# [Your site title]
Go to https://yourforum.com/llms-full.txt
```
The plugin also generates per-category, per-topic, and per-tag files that are not affected.
**I uploaded a file but `/llms.txt` returns 404**
If the uploaded file cannot be read from storage (for example, it is missing from an external object store), Discourse returns 404 and does not fall back to the generated document. Re-upload the file to resolve this.
**My site requires login, how does this behave?**
The generated document still works: it contains the account-required notice and omits public discovery links. A custom upload is always served unchanged. If you want agents to access content, consider how your login settings affect crawlability.
### **FAQs**
**Does Discourse automatically list all my topics in `llms.txt`?**
The generated default intentionally does not list individual topics. It is a minimal map that describes your site, states access policies, and points agents to the Discourse MCP interface and a few discovery routes.
**Which language is the generated document in?**
It is generated in your site’s default locale (the `default_locale` setting), regardless of the request language.
**Can I opt out?**
Yes. Set the change to **No one** in **Admin → Upcoming Changes**. Alternatively, upload a custom file with your own content.
**Why does the generated document reference Discourse MCP?**
The Discourse MCP server provides permission-aware tools for topics, posts, search, users, categories, and supported community actions. It is the preferred way for agents to work with your community, so a local MCP setup is linked in the “Preferred agent interface” section when available.
### **Additional resources**
- [Automatically generated llms.txt](https://meta.discourse.org/t/automatically-generated-llms-txt/411435)
- [Upcoming Changes](https://meta.discourse.org/t/upcoming-changes/395390)
- [Discourse SEO overview (sitemap / robots.txt)](https://meta.discourse.org/t/discourse-seo-overview-sitemap-robots-txt/280739)
---
### Author:  [@pacharanero](https://meta.discourse.org/u/pacharanero)
#### Post date: [April 8, 2026, 1:51pm UTC](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088/7 "2026-04-08T13:51:19Z")
Is it not possible for Discourse to _create_ the `llms.txt` dynamically for the site? That seems like it would be a far more useful feature, and would bring it into line with the [🤖 Discourse llms.txt Generator Plugin - #2 by Ivan\_Rapekas](https://meta.discourse.org/t/discourse-llms-txt-generator-plugin/388170/2) which this new feature collides with, overriding the path `/llms.txt` and returning a 404 even if the plugin is correctly configured.
Is this addition to the feature on the roadmap somewhere?
---
### Author:  [@pacharanero](https://meta.discourse.org/u/pacharanero)
#### Post date: [April 10, 2026, 9:48am UTC](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088/8 "2026-04-10T09:48:01Z")
My workaround for this issue was:
- Create a `llms.txt` file in VS Code containing only
- Upload this to the `admin/config/security?filter=LLMs%20TXT` Discourse core `llms.txt` section. Save.
- Test `https:///llms.txt` displays the contents of the text file above.
- Now hopefully LLMs arriving at `llms.txt` will be guided to navigate to the `llms-full.txt` which is generated dynamically by [🤖 Discourse llms.txt Generator Plugin](https://meta.discourse.org/t/discourse-llms-txt-generator-plugin/388170)
---
### Author:  [@dax](https://meta.discourse.org/u/dax)
#### Post date: [September 2, 2026, 6:18pm UTC](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088/9 "2026-09-02T18:18:02Z")
Guide updated to reflect the changes announced in [Automatically generated llms.txt](https://meta.discourse.org/t/automatically-generated-llms-txt/411435).