How to configure Discourse to use a locally installed LLM?

[meglio (Anton)]

Sorry guys I couldn’t figure out from this article if locally installed LLM can be confgirued through the standard settings UI?

[mcwumbly (Dave McClure)]

I think as long as it exposes a supported API it should be possible.

Is there a particular LLM you are planning to (or already have) installed locally?

[Lilly]

this topic may help

Self-Hosting an OpenSource LLM for DiscourseAI

The Discourse AI plugin has many features that require an LLM to be enabled, like, for example, Summarization, AI Helper, AI Search, AI Bot. While you can use a third party API, like Configure API Keys for OpenAI or Configure API Keys for Anthropic we built Discourse AI since first day to not be locked into those. Running with HuggingFace TGI HuggingFace provides an awesome container image that can get you running quickly. For example: mkdir -p /opt/tgi-cache docker run --rm --gpus all --shm-…

[meglio (Anton)]

@mcwumbly

Is there a particular LLM you are planning to (or already have) installed locally?

That’s another exercise to choose the right one - I wasn’t certain even after reading your AI-related articles here at Meta.

I guess some Open Source LLM Selector tool from the Discourse Team would be very helpful - because you know the internals and what exactly LLM must be capable of doing for it to excel in various types of tasks relevant to Discourse communities. So, the tool/wizard/LLM would ask questions or let me check on/off in a list of 20+ typical tasks I’d like the LLM to do in my community, and then get a recommended Top 3 (uncompromising but heaviest and requires expensive hardware; balanced that requires medium-priced dedicated server; and lightweight for basic tasks in small-to-medium communities that can run on a $20-40 VPS).

[pfaffman (Jay Pfaffman)]

@meglio

So, the tool/wizard/LLM would ask questions or let me check on/off in a list of 20+ typical tasks I’d like the LLM to do in my community, and then get a recommended Top 3 (uncompromising but heaviest and requires expensive hardware; balanced that requires medium-priced dedicated server; and lightweight for basic tasks in small-to-medium communities that can run on a $20-40 VPS).

I think maintaining the correct answer to that would be a full time job.

[sam (Sam Saffron)]

If you are looking for cheap, Gemini Free tier is the way to go:

Gemini Developer API Pricing  |  Gemini API  |  Google AI...

Gemini flash 2.0 is a very capable model and the free tier provides enough usage to perform quite a few things on your forum.

[meglio (Anton)]

Looking for locally hosted because I can’t send any of our content to any service outside our servers.

Re cheap options, thanks for the hint!

[Falco]

@meglio

a recommended Top 3 (uncompromising but heaviest and requires expensive hardware; balanced that requires medium-priced dedicated server; and lightweight for basic tasks in small-to-medium communities that can run on a $20-40 VPS).

Uncompromising

• DeepSeek V3 0324
• Qwen 3 235B A22

Balanced

• Qwen 3 32B / 30B A3B
• Llama 3.3 70B
• Qwen 2.5 70B

Lightweight

Maybe unsloth/gemma-3-4b-it-qat-GGUF · Hugging Face ? It’s hard at this level, much more economical to use a hosted API, like OpenRouter.