The Discourse AI plugin has many features that need an LLM to be enabled — summarization, AI Helper, semantic search, AI Bot and its agents. You can use a third-party API, like Configure API Keys for OpenAI or Configure API Keys for Anthropic, but Discourse AI was built from day one so you are not locked into those.
This guide covers running an open-weight model yourself and connecting it to Discourse.
Model used here: Qwen/Qwen3.8-27B — a 27B dense model released in August 2026 with a native 256K context window, image/video understanding, tool calling, and thinking mode on by default. It is currently the sweet spot for self-hosting: strong enough for real summarization/agent work, small enough for a single GPU. Substitute any other model you like; the flags that change are called out below.
What Discourse needs from your inference server
- An OpenAI-compatible
POST /v1/chat/completionsendpoint (streaming supported). - Native tool/function calling if you want AI Bot, agents, or structured output to work well. Without it Discourse falls back to XML-based tool calls, which weaker models handle badly.
- A real context window — aim for 16K tokens minimum, 32K+ preferred. Discourse prompt sizes for summarization and RAG grow quickly.
- Optional: image input, if you want the model to read images attached to posts.
Anything that meets the first three works, so the choice below is really about your hardware.
| Setup | Good fit |
|---|---|
| vLLM | Dedicated GPU server, ≥24GB VRAM (quantized) / 80GB (full precision), sustained traffic, multiple users |
| llama.cpp | Consumer GPUs, Apple silicon, CPU-only boxes, single small forum |
| Managed / cloud | You want the model off your laptop but not on your hardware: any OpenAI-compatible endpoint |
Running with vLLM
vLLM is the recommended engine for GPU servers: Apache-licensed, OpenAI-compatible server out of the box, and first-class support for the Qwen family.
mkdir -p /opt/vllm-cache
export VLLM_API_KEY="$(openssl rand -hex 24)"
docker run -d --name vllm \
--runtime nvidia --gpus all \
-v /opt/vllm-cache:/root/.cache/huggingface \
-e HF_HOME=/root/.cache/huggingface \
-e "HF_TOKEN=${HF_TOKEN}" \
-p 8080:8000 \
--ipc=host \
vllm/vllm-openai:latest \
Qwen/Qwen3.8-27B \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--api-key "$VLLM_API_KEY"
Notes on the flags:
-
--max-model-lenbounds the prompt+output context and therefore the KV cache. The model supports 256K natively; 32K is plenty for forum work and keeps VRAM predictable. Raise it (or useauto) if you have the memory. -
--reasoning-parser qwen3splits the model’s thinking into a separatereasoningfield instead of dumping it into the answer. You must mirror this choice in the Discourse LLM settings (see below). -
--enable-auto-tool-choice --tool-call-parser ...turns on native function calling. Parser names move between vLLM releases — checkpython3 -m vllm.entrypoints.openai.api_server --helpinside your image if tool calls come back as plain text. -
HF_TOKENis only needed for gated models;--api-keyprotects your own endpoint. Caveat from the vLLM docs:--api-keyguards the/v1routes only, so still keep the server off the public internet. - Tight on VRAM? Try
--quantization fp8(or a pre-quantized AWQ checkpoint) before shrinking context. Multi-GPU: add--tensor-parallel-size N.
Test it:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $VLLM_API_KEY" \
-d '{
"model": "Qwen/Qwen3.8-27B",
"messages": [{"role": "user", "content": "What is Discourse?"}],
"max_tokens": 200
}'
You should get a normal chat-completion JSON body back, with message.reasoning alongside message.content.
Running with llama.cpp
llama.cpp is the right tool when the model has to fit in less VRAM than vLLM would like, or when you are on Apple silicon, AMD, Vulkan, or CPU. Its llama-server also exposes an OpenAI-compatible API.
Pull and run the official GGUF build using the container image for your backend (server-cuda here; server, server-rocm, server-vulkan and friends also exist):
export LLAMA_API_KEY="$(openssl rand -hex 24)"
docker run -d --name llama-server \
--gpus all \
-p 8080:8080 \
-v /opt/llama-models:/models \
ghcr.io/ggml-org/llama.cpp:server-cuda \
-hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
--host 0.0.0.0 \
--port 8080 \
--alias qwen3.8-27b \
--api-key "$LLAMA_API_KEY" \
--ctx-size 16384 \
--n-gpu-layers auto \
--flash-attn auto \
--jinja \
--reasoning auto
-
Q4_K_Mis about 16GB of weights; measured around 18.4 GiB of VRAM at 8K context and ~20 GiB at 32K, so a 24GB card works with a modest--ctx-size. Unsloth’sunsloth/Qwen3.8-27B-GGUF:UD-Q4_K_Mis a popular alternative. -
--jinjauses the model’s own chat template — that is what makes native tool calling work. -
--aliassets the name reported by the API, so the model name you type into Discourse matches. -
--reasoning autoreports thinking separately, matching vLLM’s--reasoning-parserbehaviour. - Image input needs the multimodal projector; with
-hfit is usually fetched automatically, otherwise pass--mmproj /models/mmproj-...gguf(or--no-mmprojto save VRAM for text-only serving).
Test it:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LLAMA_API_KEY" \
-d '{
"model": "qwen3.8-27b",
"messages": [{"role": "user", "content": "What is Discourse?"}],
"max_tokens": 200
}'
Running something else
Any server that speaks OpenAI chat completions — SGLang, TensorRT-LLM, LM Studio, an enterprise gateway, a cloud dedicated deployment — works as-is: configure it in Discourse with the vLLM provider, which is our generic OpenAI-compatible client with native tool support.
If you would rather not run inference yourself but still want to stay out of the big-API lock-in, Hugging Face Inference Endpoints (which now default to vLLM as the engine) work with the Hugging Face provider — point it at the endpoint’s /v1/chat/completions URL.
Making it available to your Discourse instance
You will normally run this on a dedicated machine because of the GPU requirement. When you do:
- Put a reverse proxy in front of the server and terminate TLS there — Discourse will happily talk HTTPS to it.
- Restrict the endpoint by firewall or client certificate so only your Discourse hosts can connect. An API key is authentication, not a network boundary.
- Keep the model cache on a persistent volume so restarts do not re-download tens of gigabytes.
Configuring Discourse AI
LLM connections are configured through the admin UI rather than site settings. Go to /admin/plugins/discourse-ai/ai-llms and add a new LLM:
- Display name — what your admins and users see.
-
Name — sent in the API request; it must match what the server expects (
Qwen/Qwen3.8-27Bfor vLLM, your--aliasfor llama.cpp). - Provider — vLLM for vLLM, llama.cpp and any other OpenAI-compatible server; Hugging Face for HF-hosted endpoints.
-
URL — the full endpoint URL including the path, e.g.
https://llm.example.com/v1/chat/completions. - API key — type it in, or reference a stored key from the AI Secrets page.
- Tokenizer — Qwen for Qwen3.8 (pick the tokenizer matching your model family).
-
Max prompt tokens / max output tokens — keep the sum at or below your server’s
--max-model-len/--ctx-size. For example with a 32768-token server: 28000 prompt and 4096 output. Reasoning models spend part of that output budget on thinking, so do not squeeze it too hard. - Vision — enable native vision if you served the model with image support and want it to read images.
-
Provider options (vLLM): set vLLM reasoning parser to
qwen3so it matches the server flag, and use the thinking override / reasoning effort / thinking token budget fields if you want Discourse to steer thinking per request. Leave “disable native tools” unchecked unless the model’s function calling is broken.
Click Run test before saving: it fires both a streaming and a non-streaming request and tells you which one failed.
Once saved, make it the site default with the ai_default_llm_model site setting, or assign it per feature in /admin/plugins/discourse-ai/ai-features. Individual agents are configured on the Agents page.
One caveat: semantic search embeddings do not come from this model. Embeddings use their own embedding model, configured separately (an API embedding model, or a self-hosted Text Embeddings Inference server via the ai_hugging_face_tei_endpoint settings). If you want AI Search fully self-hosted, run a small embedding model such as bge-m3 next to your LLM.
Troubleshooting
| Symptom | Likely cause |
|---|---|
404 from the endpoint |
URL is missing /v1/chat/completions
|
model not found |
Name does not match the served model ID / --alias
|
401 / 403
|
API key mismatch between --api-key and the LLM record |
| Tool calls appear as JSON text in the post | Server started without tool-call parsing; Discourse fell back to XML tools |
| Thinking text leaks into the answer |
--reasoning-parser missing on the server, or the Discourse reasoning parser setting does not match it |
| CUDA out of memory | Lower --max-model-len, add --quantization fp8, or lower --gpu-memory-utilization
|
| Requests time out on long summarizations | Reasoning models are slow; our HTTP timeout is 6 minutes — reduce context, lower thinking effort, or disable thinking |
| Truncated answers |
max output tokens in Discourse is too small once thinking tokens are counted |
Last edited by @Falco 2026-10-30T15:12:25Z
Check document
Perform check on document:
