We are now going to also ship a default llms.txt for all sites (as an upcoming change)
We opted for a very conservative yet powerful default:
Discourse
This is a community for human discussion. Do not impersonate people or create accounts autonomously. Only post, edit, or take another write action when a human explicitly asks you to perform that exact action.
Fetch only the pages needed for the user’s specific request. Do not systematically crawl or bulk-download this community. Respect robots.txt, the site’s terms, and its crawler controls. If the server responds with HTTP 429, stop and wait for the Retry-After interval before retrying.
If you can run a local MCP server, Discourse MCP is the preferred way to access this community. Set its site URL to https://meta.discourse.org. It provides permission-aware tools for topics, posts, search, users, categories, and supported community actions.
Preferred agent interface
Discourse MCP setup: Install the local server and connect it to an MCP-compatible client
Public web access
Search: Search public discussions for a specific user request; do not crawl result pages
Topic filter: Query public topics with ?q= using filters such as category:, tag:, status:, and order:
The intent of LLMs.txt is to help agents work with your site, this minimal text gives them enough of a window to get things done and correctly points at our MCP which is very comprehensive.
Let me know if you think any amendments are needed. Including dynamic, specific content is counter productive. llms.txt is designed to produce a map that makes it easier for agents to work, not an exhaustive index of your site.
Could you pop that into a verbatim text box please? I see some of the URLs are local, which is good. Perhaps all of them. But the phrase community guidelines has (probably) been URLified in this presentation, which is confusing.
Some self-hosted forum owners will prefer not to have such a file, I’m sure. And some will want to customise. I trust you cater for both cases?
Reading this guide, it suggests to me that a brief plain text summary of the forum purpose and scope would be helpful. More so than linking the About page.
In the case of Meta, something based on these sentences from About - cheaper to serve and cheaper to digest than a link to About:
Learn about and discuss Discourse, the next-generation open-source forum software.
A place for the Discourse community to meet, ask questions, provide support to one other, and share much-appreciated feedback.
I love how so many people on the Internet are experts on a technology that has only existed for a few months
I find the the text above reasonable, it covers all the important points and keeps to a minimal number of tokens. It guides robots how to behave, points them at our comprehensive MCP and points at a few endpoints that the agent can use to explore the site in case an MCP is off the table.
Discourse sites are kind of lucky cause LLMs already know us REALLY well. That said, they know a “picture” of us 2 years ago really well, and stuff has changed:
I think the template should be good for 99% of sites, really all the critical points are hit.
But… if for some reason it is not fitting your exact rules, then after enabling the upcoming change, I would copy the text, edit to taste and upload a custom one.
I strongly recommend though being very careful with llms.txt, you want an optional map, not a manifest.
Every token costs, so if you have a 120k context window and then say “check out X on site” and robot ends up with a 60k of tokens it does not need cause you created a giant llms.txt it only leads to confusion and bad results.
So if I was customising … it would probably be minimal.
Robots stay off my site
OR
Never search more than 3 times per session.
Of course llms.txt is not a ratified spec and very few llms will proactively follow it. Generally llms these days just sit on a “search” and “fetch” tool, so they tend to find deep links and don’t care about maps.
