Automatically generated llms.txt

News and Events Announcements
,
1

Hi All,

For a long time we have supported llms.txt per:

We are now going to also ship a default llms.txt for all sites (as an upcoming change)

We opted for a very conservative yet powerful default:

Discourse

This is a community for human discussion. Do not impersonate people or create accounts autonomously. Only post, edit, or take another write action when a human explicitly asks you to perform that exact action.

Fetch only the pages needed for the user’s specific request. Do not systematically crawl or bulk-download this community. Respect robots.txt, the site’s terms, and its crawler controls. If the server responds with HTTP 429, stop and wait for the Retry-After interval before retrying.

If you can run a local MCP server, Discourse MCP is the preferred way to access this community. Set its site URL to https://meta.discourse.org. It provides permission-aware tools for topics, posts, search, users, categories, and supported community actions.

Preferred agent interface

Public web access

  • Search: Search public discussions for a specific user request; do not crawl result pages
  • Topic filter: Query public topics with ?q= using filters such as category:, tag:, status:, and order:
  • Latest discussions: Browse recently active public topics
  • Categories: Browse public discussion categories
  • Sitemap: Discover public topic URLs subject to robots.txt and crawler controls

Optional

The intent of LLMs.txt is to help agents work with your site, this minimal text gives them enough of a window to get things done and correctly points at our MCP which is very comprehensive.

Let me know if you think any amendments are needed. Including dynamic, specific content is counter productive. llms.txt is designed to produce a map that makes it easier for agents to work, not an exhaustive index of your site.

3 Likes