sam announced that Discourse will soon ship a default llms.txt file for all sites, building on existing support. The conservative default aims to guide AI agents effectively by prohibiting impersonation and bulk crawling, while encouraging the use of the Discourse MCP server for comprehensive, permission-aware access. It also provides minimal web access endpoints like search and filters. sam emphasized that llms.txt should serve as a map for agents, not an exhaustive index, warning against including dynamic content or excessive details that consume valuable context windows.
Ed_S raised concerns about URL formatting in the announcement and questioned whether self-hosted owners could disable or customize the file. sam clarified that the file is 100% customizable via upload, recommending a simple “Robots not welcome” text file for those wishing to opt out entirely. Ed_S suggested that a brief plain text summary of the forum’s purpose would be more efficient than linking to the About page, a point sam acknowledged but countered by noting that LLMs already have strong existing knowledge of Discourse, though this data may be outdated regarding recent features like MCP.
Regarding customization, sam advised that the default template suits most sites, but users should edit it minimally if needed to avoid confusing agents with oversized files. For those seeking to block agents completely, NateDhaliwal provided instructions from official documentation: users can upload a custom llms.txt (up to 512 KB) via Admin → Settings → Security → LLMs TXT, which overrides the generated version. This allows site owners to mirror existing policies or enforce stricter rules, such as limiting search queries per session.
We are now going to also ship a default llms.txt for all sites (as an upcoming change)
We opted for a very conservative yet powerful default:
Discourse
This is a community for human discussion. Do not impersonate people or create accounts autonomously. Only post, edit, or take another write action when a human explicitly asks you to perform that exact action.
Fetch only the pages needed for the user’s specific request. Do not systematically crawl or bulk-download this community. Respect robots.txt, the site’s terms, and its crawler controls. If the server responds with HTTP 429, stop and wait for the Retry-After interval before retrying.
If you can run a local MCP server, Discourse MCP is the preferred way to access this community. Set its site URL to https://meta.discourse.org. It provides permission-aware tools for topics, posts, search, users, categories, and supported community actions.
Preferred agent interface
Discourse MCP setup: Install the local server and connect it to an MCP-compatible client
Public web access
Search: Search public discussions for a specific user request; do not crawl result pages
Topic filter: Query public topics with ?q= using filters such as category:, tag:, status:, and order:
The intent of LLMs.txt is to help agents work with your site, this minimal text gives them enough of a window to get things done and correctly points at our MCP which is very comprehensive.
Let me know if you think any amendments are needed. Including dynamic, specific content is counter productive. llms.txt is designed to produce a map that makes it easier for agents to work, not an exhaustive index of your site.
Could you pop that into a verbatim text box please? I see some of the URLs are local, which is good. Perhaps all of them. But the phrase community guidelines has (probably) been URLified in this presentation, which is confusing.
Some self-hosted forum owners will prefer not to have such a file, I’m sure. And some will want to customise. I trust you cater for both cases?
Reading this guide, it suggests to me that a brief plain text summary of the forum purpose and scope would be helpful. More so than linking the About page.
In the case of Meta, something based on these sentences from About - cheaper to serve and cheaper to digest than a link to About:
Learn about and discuss Discourse, the next-generation open-source forum software.
A place for the Discourse community to meet, ask questions, provide support to one other, and share much-appreciated feedback.
I love how so many people on the Internet are experts on a technology that has only existed for a few months
I find the the text above reasonable, it covers all the important points and keeps to a minimal number of tokens. It guides robots how to behave, points them at our comprehensive MCP and points at a few endpoints that the agent can use to explore the site in case an MCP is off the table.
Discourse sites are kind of lucky cause LLMs already know us REALLY well. That said, they know a “picture” of us 2 years ago really well, and stuff has changed:
I think the template should be good for 99% of sites, really all the critical points are hit.
But… if for some reason it is not fitting your exact rules, then after enabling the upcoming change, I would copy the text, edit to taste and upload a custom one.
I strongly recommend though being very careful with llms.txt, you want an optional map, not a manifest.
Every token costs, so if you have a 120k context window and then say “check out X on site” and robot ends up with a 60k of tokens it does not need cause you created a giant llms.txt it only leads to confusion and bad results.
So if I was customising … it would probably be minimal.
Robots stay off my site
OR
Never search more than 3 times per session.
Of course llms.txt is not a ratified spec and very few llms will proactively follow it. Generally llms these days just sit on a “search” and “fetch” tool, so they tend to find deep links and don’t care about maps.
I’m a bit late on this, but where can I set a custom one? I have no interest in “agents” bombarding my site with requests and would like to be able to mirror my root domain’s version.
