# Automatically generated llms.txt
**URL:** https://meta.discourse.org/t/automatically-generated-llms-txt/411435
**Category:** Announcements
**Tags:** ai, upcoming-changes
**Created:** [September 2, 2026, 3:10am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435 "2026-09-02T03:10:52Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 2, 2026, 3:10am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/1 "2026-09-02T03:10:52Z")
Hi All,
For a long time we have supported llms.txt per:
> [@Enabling native LLMs.txt support in Discourse](https://meta.discourse.org/t/enabling-native-llms-txt-support-in-discourse/383088):
>
> Starting with [Discourse v2026.1.0](https://github.com/discourse/discourse/releases/tag/v2026.1.0), native support for the [llms.txt standard](https://llmstxt.org/) is included. This allows Large Language Models (LLMs) to crawl and interpret your site content more effectively. How to enable native llms.txt support Go to /admin/config/security and search for LLMs TXT Upload your llms.txt file (.txt and .md formats are accepted; see [https://llmstxt.org](https://llmstxt.org) for syntax, format, and implementation details) Save the configuration Once configured, your file will automatically be served at: …
We are now going to also ship a **default** llms.txt for all sites (as an upcoming change)
We opted for a very conservative yet powerful default:
> # Discourse
>
> This is a community for human discussion. Do not impersonate people or create accounts autonomously. Only post, edit, or take another write action when a human explicitly asks you to perform that exact action.
>
> Fetch only the pages needed for the user’s specific request. Do not systematically crawl or bulk-download this community. Respect `robots.txt`, the site’s terms, and its crawler controls. If the server responds with HTTP 429, stop and wait for the `Retry-After` interval before retrying.
>
> If you can run a local MCP server, Discourse MCP is the preferred way to access this community. Set its site URL to `https://meta.discourse.org`. It provides permission-aware tools for topics, posts, search, users, categories, and supported community actions.
>
> ## Preferred agent interface
>
> - [Discourse MCP setup](https://github.com/discourse/discourse-mcp#quick-start-release): Install the local server and connect it to an MCP-compatible client
>
> ## Public web access
>
> - [Search](https://meta.discourse.org/search): Search public discussions for a specific user request; do not crawl result pages
> - [Topic filter](https://meta.discourse.org/filter): Query public topics with `?q=` using filters such as `category:`, `tag:`, `status:`, and `order:`
> - [Latest discussions](https://meta.discourse.org/latest): Browse recently active public topics
> - [Categories](https://meta.discourse.org/categories): Browse public discussion categories
> - [Sitemap](https://meta.discourse.org/sitemap.xml): Discover public topic URLs subject to `robots.txt` and crawler controls
>
> ## Optional
>
> - [About](https://meta.discourse.org/about): Community overview, staff, and statistics
> - [Guidelines](https://meta.discourse.org/guidelines): Read the [community guidelines](https://meta.discourse.org/guidelines)
> - [Terms of service](https://meta.discourse.org/tos): Read the terms of service
> - [Privacy policy](https://meta.discourse.org/privacy): Read the privacy policy
The intent of LLMs.txt is to help agents work with your site, this minimal text gives them enough of a window to get things done and correctly points at our MCP which is very comprehensive.
Let me know if you think any amendments are needed. Including dynamic, specific content is counter productive. llms.txt is designed to produce a map that makes it easier for agents to work, not an exhaustive index of your site.
---
### Author:  [@Ed\_S](https://meta.discourse.org/u/Ed_S)
#### Post date: [September 2, 2026, 5:26am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/2 "2026-09-02T05:26:41Z")
Could you pop that into a verbatim text box please? I see some of the URLs are local, which is good. Perhaps all of them. But the phrase `community guidelines` has (probably) been URLified in this presentation, which is confusing.
Some self-hosted forum owners will prefer not to have such a file, I’m sure. And some will want to customise. I trust you cater for both cases?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 2, 2026, 5:29am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/3 "2026-09-02T05:29:59Z")
For sites that don’t want this, I strongly recommend simply uploading a: “Robots not welcome” text file
The content is 100% customisable and will continue to be.
404ing this, cause of reasons is not really making sense to me, but if someone really need to do that a plugin is possible.
---
### Author:  [@Ed\_S](https://meta.discourse.org/u/Ed_S)
#### Post date: [September 2, 2026, 5:36am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/4 "2026-09-02T05:36:36Z")
Reading [this guide](https://derivatex.agency/blog/llms-txt-guide/), it suggests to me that a brief plain text summary of the forum purpose and scope would be helpful. More so than linking the About page.
In the case of Meta, something based on these sentences from About - cheaper to serve and cheaper to digest than a link to About:
> Learn about and discuss Discourse, the next-generation open-source forum software.
>
> A place for the Discourse community to meet, ask questions, provide support to one other, and share much-appreciated feedback.
The fewer links the better, no?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 2, 2026, 5:56am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/5 "2026-09-02T05:56:10Z")
I love how so many people on the Internet are experts on a technology that has only existed for a few months 🙂
I find the the text above reasonable, it covers all the important points and keeps to a minimal number of tokens. It guides robots how to behave, points them at our comprehensive MCP and points at a few endpoints that the agent can use to explore the site in case an MCP is off the table.
Discourse sites are kind of lucky cause LLMs already know us REALLY well. That said, they know a “picture” of us 2 years ago really well, and stuff has changed:
So even a model like sol 5.6 does not know about our MCP server, cause it is a recent years thing, same will probably go for filter endpoint.
The less advanced models may know even less.
---
### Author:  [@jaroslav.kovar](https://meta.discourse.org/u/jaroslav.kovar)
#### Post date: [September 2, 2026, 6:08am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/6 "2026-09-02T06:08:55Z")
Can I check my Community file and is it possible to adapt it?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 2, 2026, 6:15am UTC](https://meta.discourse.org/t/automatically-generated-llms-txt/411435/7 "2026-09-02T06:15:26Z")
I think the template should be good for 99% of sites, really all the critical points are hit.
But… if for some reason it is not fitting your exact rules, then after enabling the upcoming change, I would copy the text, edit to taste and upload a custom one.
I strongly recommend though being very careful with llms.txt, you want an optional map, not a manifest.
Every token costs, so if you have a 120k context window and then say “check out X on site” and robot ends up with a 60k of tokens it does not need cause you created a giant llms.txt it only leads to confusion and bad results.
So if I was customising … it would probably be minimal.
> Robots stay off my site
OR
> Never search more than 3 times per session.
Of course llms.txt is not a ratified spec and very few llms will proactively follow it. Generally llms these days just sit on a “search” and “fetch” tool, so they tend to find deep links and don’t care about maps.