Discourse core now includes Markdown endpoints for topic lists and views

Discourse now supports native Markdown endpoints, making it easier for AI tools and other clients to read forum content without parsing full HTML pages.

This feature will be enabled by default on all hosted sites via the Upcoming Changes system. Administrators that want to opt out can do so by disabling the enable_markdown_endpoints site setting.

Many thanks to @benword for creating the original Discourse to Markdown plugin, it was the precursor to this feature in Discourse core.


The Markdown output is generated from posts’ rendered (“cooked”) HTML, preserving the content readers see, including expanded links and processed formatting. Discourse-specific elements such as quotes, oneboxes, code blocks, polls, and collapsible sections are converted back to Markdown. Topic responses include metadata and pagination links, and converted post bodies are cached using a content digest so edits produce fresh output.

Clients can request Markdown explicitly through .md URLs or by sending an Accept: text/markdown header. Negotiation respects quality values and selects Markdown when it is preferred over HTML and JSON; explicit .md requests retain their formats. Supported HTML pages advertise their Markdown equivalent through an HTTP Link header and a <link rel="alternate"> element.

20 Likes

Could you please clarify which topic lists are supported and which are not?

Since I am currently using the original Discourse-to-Markdown plugin, do I need to disable and remove it to avoid conflicting outputs? Please advise.

Works fine without the plugin and the upcoming feature enabled.

1 Like

One question: for those using Cloudflare’s proxy, it seems there’s a conflict between the core function and their conversion tool. The question is: if I’m behind the proxy, since the function is for subscribers, will they prevent the header from being converted from HTML to .md, or, because the client supports the conversion, does it happen regardless?

Yes. If the plugin stays enabled, it replaces some of the core endpoints, so we recommend disabling/uninstalling it in order to use the functionality now in core.

Currently, the following:

Supported Examples
Main lists /latest.md, /hot.md, /top.md
Personalized lists, requiring authentication /new.md, /unread.md
Default category/subcategory lists /c/support/6.md, /c/parent/child/12.md
Single-tag lists /tag/example.md, /tag/example/123.md

/categories.md and /tags.md are additionally supported as directories. Route definitions

I am not too familiar with the Cloudflare feature, but from what I can tell, it catches the Accept text/markdown request before it hits the server, converts the html and then serves that. So, it looks like that feature would override the Discourse one, if/when using Cloudflare.

3 Likes

It would override it if enabled, right? I couldn’t find any information on their blog about whether they prevent the origin itself from serving that header.

I don’t know for sure, easiest option is to test on a live site and compare the output with what meta outputs. There will be differences in what content is included. If the response you get is the same with or without Cloudflare’s feature, then it is respecting the Markdown returned by Discourse core.

1 Like

Thanks for the guidance. Here’s the response on that:

  • X-Discourse-Route: topics/show: Shows the internal Discourse (Ruby on Rails) controller/action processing the topic.
  • X-Runtime: 0.133969: The time the application took to generate the response (approx. 133ms).
  • Cf-Ray: ...-GRU: Request served by the Cloudflare edge in Guarulhos/São Paulo (GRU).
  • Cf-Cache-Status: DYNAMIC and Cache-Control: no-cache, no-store: Dynamic content that does not get stuck in edge caches.
  • Inspecting the standard HTML URL without .json, the server responds with:
    • Link: <https://segredin.com/t/conselhos-duvidosos/22054.md>; rel="alternate"; type="text/markdown"
    • X-Discourse-Crawler-View: true (indicating that Discourse also provides a clean .md / native Markdown version for crawlers and readers).

The Link header indicating the alternate version: Link: <https://segredin.com/t/conselhos-duvidosos/22054.md>; rel="alternate"; type="text/markdown"

It returns Vary: Accept from the origin, independent of external DNS-level functions.

If the request opts to make a request without .json, it will return .md as the default conversion.

HTTP/1.1 200 OK
Content-Type: text/markdown
Vary: Accept

I was unsure because I received 123k requests from Claude, and most of them, since I updated Discourse with this core feature, have not spiked vertically. I will monitor this over the coming weeks.

Thanks, I was confused because I first tried on a topic list filtered for a category and a tag and it didn’t work. I tend to choose bad examples for testing.

Why did you include /new and /unread but not /unseen?

1 Like

Does core intend discourse-post-event blocks to have a dedicated Markdown representation, in the same way that polls, quotes, oneboxes and collapsible sections already do? Technically, adding one to CookedProcessor looks quite feasible: detect div.discourse-post-event, read its data-* attributes, and replace it with a preserved Markdown block before the generic ReverseMarkdown pass.

2 Likes

Thank you for that suggestion, it’s implemented in this PR that will be merged shortly.

2 Likes