How to configure Discourse to use a locally installed LLM?

Product Support

AI-generated summary

In a Support discussion titled How to configure Discourse to use a locally installed LLM?, users explore whether locally hosted LLMs can be configured via Discourse’s standard UI. meglio seeks clarification after reading an article, while mcwumbly suggests it’s possible if the LLM exposes a supported API, asking which model meglio plans to use. Lilly shares a helpful topic on self-hosting open-source LLMs. meglio proposes a hypothetical Open Source LLM Selector tool from the Discourse team that would recommend models based on community needs and hardware constraints — categorized as uncompromising, balanced, or lightweight. pfaffman humorously notes maintaining such a tool would be a full-time job. sam recommends Gemini’s free tier for cost-effective usage, but meglio clarifies the need for local hosting due to data privacy. Falco offers concrete model suggestions: Uncompromising (DeepSeek V3 0324, Qwen 3 235B), Balanced (Qwen 3 32B, Llama 3.3 70B), and Lightweight (unsloth/gemma-3-4b-it-qat-GGUF), noting hosted APIs like OpenRouter may be more economical at the lightweight tier. The thread reflects a practical, privacy-conscious approach to integrating LLMs into Discourse communities.

1

Sorry guys I couldn’t figure out from this article if locally installed LLM can be confgirued through the standard settings UI?

2

I think as long as it exposes a supported API it should be possible.

Is there a particular LLM you are planning to (or already have) installed locally?

3

this topic may help

4

That’s another exercise to choose the right one - I wasn’t certain even after reading your AI-related articles here at Meta.

I guess some Open Source LLM Selector tool from the Discourse Team would be very helpful - because you know the internals and what exactly LLM must be capable of doing for it to excel in various types of tasks relevant to Discourse communities. So, the tool/wizard/LLM would ask questions or let me check on/off in a list of 20+ typical tasks I’d like the LLM to do in my community, and then get a recommended Top 3 (uncompromising but heaviest and requires expensive hardware; balanced that requires medium-priced dedicated server; and lightweight for basic tasks in small-to-medium communities that can run on a $20-40 VPS).

5

I think maintaining the correct answer to that would be a full time job. :rofl:

6

If you are looking for cheap, Gemini Free tier is the way to go:

Gemini flash 2.0 is a very capable model and the free tier provides enough usage to perform quite a few things on your forum.

7

Looking for locally hosted because I can’t send any of our content to any service outside our servers.

Re cheap options, thanks for the hint!

8

Uncompromising

  • DeepSeek V3 0324
  • Qwen 3 235B A22

Balanced

  • Qwen 3 32B / 30B A3B
  • Llama 3.3 70B
  • Qwen 2.5 70B

Lightweight

Maybe unsloth/gemma-3-4b-it-qat-GGUF · Hugging Face ? It’s hard at this level, much more economical to use a hosted API, like OpenRouter.