Enabling native LLMs.txt support in Discourse

Documentation Site Management
1

:bookmark: This guide explains how to make your community discoverable and usable by AI agents and LLM crawlers through the llms.txt standard. It covers both the automatically generated default document and custom uploaded files.

:person_raising_hand: Required user level: Administrator

Summary

Discourse includes native support for the llms.txt standard, a proposed convention for providing an LLM-friendly overview of a website. Think of it as “robots.txt for AI”: a small Markdown file at /llms.txt that helps large language models (LLMs) and AI agents understand what your site is about and how to access it responsibly.

You can choose between two approaches, or combine them:

  1. Let Discourse generate a default document automatically from your site settings
  2. Upload your own custom llms.txt file for full control over the content

A custom upload always takes priority over the generated document.

What is llms.txt?

llms.txt is a proposed standard that provides a curated, agent-friendly map of a website. It is served at the site root (/llms.txt) and follows a precise Markdown format:

  • An H1 heading with the name of the site (the only required section)
  • A blockquote with a short summary
  • Optional markdown sections with more detail
  • H2-delimited sections containing lists of useful links

Unlike sitemap.xml, which lists pages for search engines, llms.txt is a concise overview designed to fit in an LLM context window. The detail lives behind the links, which are fetched only when needed.

Starting with Discourse v2026.1.0, Discourse can serve llms.txt from the site root.

How Discourse serves /llms.txt

Discourse handles requests to /llms.txt in this order:

  1. If you have uploaded a custom file, Discourse serves it unchanged as text/plain
  2. If no custom file is uploaded and the generated default is enabled, Discourse generates and serves the document
  3. Otherwise, Discourse returns a 404

The generated document is identical for anonymous visitors and signed-in users, and it is not affected by login required redirects.

The generated default document

When no custom file is uploaded and the feature is enabled, Discourse builds a concise document using your site’s existing settings and the site’s default locale. It includes:

  • The site title (from the title setting), with the hostname as fallback
  • A summary blockquote using your site_description, or short_site_description as a fallback
  • A human community policy stating that the site is for human discussion and that agents must only write when a human explicitly asks them to
  • An agent access policy asking agents to fetch only what they need, respect robots.txt and crawler controls, and honor HTTP 429 with the Retry-After interval
  • A Discourse MCP reference, pointing agents to the Discourse MCP server as the preferred, permission-aware interface
  • A Preferred agent interface section with the Discourse MCP setup link
  • A Public web access section with links to /search, /filter, /latest, and /categories, plus /sitemap.xml when the enable_sitemap setting is on
  • An Optional section with links to /about, /guidelines, /tos, and /privacy

If your site has login_required enabled, the document adds a note that content is available only to authenticated members and omits the public discovery links described above. Links are base-path aware, so they work correctly on subfolder installs.

You can see the announcement and discuss the generated template in Automatically generated llms.txt. Meta itself currently serves a stock file.

Enable the generated default document

The generated default is delivered through the Upcoming Changes system as a beta change that admins can opt into or out of.

  1. Go to Admin → Configure → Upcoming Changes, or visit /admin/config/upcoming-changes
  2. Find the change that generates a default /llms.txt when no custom file is uploaded
  3. In the Enabled for dropdown, choose Everyone to enable it for all visitors
  4. To turn it off, choose No one

Promoted changes appear here automatically when they are ready, and admins are notified on the dashboard. If you prefer that no llms.txt file is served at all, set this change to No one or upload a minimal custom file.

Upload a custom llms.txt

If you prefer to control the content exactly, you can upload your own file.

  1. Prepare a llms.txt file in .txt or .md format (maximum file size is 512 KB)
  2. Go to Admin → Settings → Security, or visit /admin/config/security, and search for LLMs TXT
  3. Upload your file and save the configuration

Once configured, Discourse serves your file unchanged at:

https://yourforum.com/llms.txt

Refer to llmstxt.org for syntax and format guidance. A custom upload always overrides the generated document.

Verify your setup

You can check what agents will see by requesting the file in a browser or with curl:

curl https://yourforum.com/llms.txt

Best practices

  • Keep the file concise. llms.txt is a map, not a manifest; every token costs context, and a huge file leads to confused agents and worse results
  • Only serve the generated default if its content matches your policies. If you need different wording, copy the generated text, edit it to suit your community, and upload it as a custom file
  • Structure custom files per the spec: H1 title, blockquote summary, then H2 sections with link lists, using an Optional section for secondary links
  • Combine llms.txt with crawler controls. To tune automated access, configure the slow_down_crawler_user_agents and slow_down_crawler_rate settings (Admin → Settings → Security)
  • Remember that llms.txt describes your site, while robots.txt controls access. They serve different purposes and complement each other

Common issues and solutions

I get a 404 at /llms.txt

This happens when no custom file is uploaded and the generated default is not enabled. Upload a custom file, enable the generated default in Admin → Upcoming Changes, or both.

I use a llms.txt Generator plugin and the main file stopped working

Core Discourse serves the /llms.txt path, so it takes precedence over plugins that generate it. If you rely on the plugin’s dynamically generated files, a practical workaround is to upload a minimal custom llms.txt that points agents to the plugin-generated content, for example:

# [Your site title]

Go to https://yourforum.com/llms-full.txt

The plugin also generates per-category, per-topic, and per-tag files that are not affected.

I uploaded a file but /llms.txt returns 404

If the uploaded file cannot be read from storage (for example, it is missing from an external object store), Discourse returns 404 and does not fall back to the generated document. Re-upload the file to resolve this.

My site requires login, how does this behave?

The generated document still works: it contains the account-required notice and omits public discovery links. A custom upload is always served unchanged. If you want agents to access content, consider how your login settings affect crawlability.

FAQs

Does Discourse automatically list all my topics in llms.txt?

The generated default intentionally does not list individual topics. It is a minimal map that describes your site, states access policies, and points agents to the Discourse MCP interface and a few discovery routes.

Which language is the generated document in?

It is generated in your site’s default locale (the default_locale setting), regardless of the request language.

Can I opt out?

Yes. Set the change to No one in Admin → Upcoming Changes. Alternatively, upload a custom file with your own content.

Why does the generated document reference Discourse MCP?

The Discourse MCP server provides permission-aware tools for topics, posts, search, users, categories, and supported community actions. It is the preferred way for agents to work with your community, so a local MCP setup is linked in the “Preferred agent interface” section when available.

Additional resources

14 Likes
🤖 Discourse llms.txt Generator Plugin
Staff category bubble visible
Automatically generated llms.txt
7

Is it not possible for Discourse to create the llms.txt dynamically for the site? That seems like it would be a far more useful feature, and would bring it into line with the 🤖 Discourse llms.txt Generator Plugin - #2 by Ivan_Rapekas which this new feature collides with, overriding the path /llms.txt and returning a 404 even if the plugin is correctly configured.

Is this addition to the feature on the roadmap somewhere?

5 Likes
🤖 Discourse llms.txt Generator Plugin
8

My workaround for this issue was:

  • Create a llms.txt file in VS Code containing only

    Go to [https://<YOUR-BASEURL>/llms-full.txt](https://<YOUR-BASEURL>/llms-full.txt)

  • Upload this to the admin/config/security?filter=LLMs%20TXT Discourse core llms.txt section. Save.

  • Test https://<YOUR-BASEURL>/llms.txt displays the contents of the text file above.

  • Now hopefully LLMs arriving at llms.txt will be guided to navigate to the llms-full.txt which is generated dynamically by 🤖 Discourse llms.txt Generator Plugin

1 Like
9

Guide updated to reflect the changes announced in Automatically generated llms.txt.

3 Likes