### Author:  [@Discourse](https://meta.discourse.org/u/Discourse)
#### Post date: [January 8, 2024, 8:39pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/1 "2024-01-08T20:39:55Z")
The [Discourse AI](https://meta.discourse.org/t/discourse-ai/259214) plugin has many features that require an LLM to be enabled, like, for example, Summarization, AI Helper, AI Search, AI Bot. While you can use a third party API, like [Configure API Keys for OpenAI](https://meta.discourse.org/t/configure-api-keys-for-openai/280783) or [Configure API Keys for Anthropic](https://meta.discourse.org/t/configure-api-keys-for-anthropic/280936) we built Discourse AI since first day to not be locked into those.
### Running with HuggingFace TGI
HuggingFace provides an awesome container image that can get you running quickly.
For example:
```plaintext
mkdir -p /opt/tgi-cache
docker run --rm --gpus all --shm-size 1g -p 8080:80 \
-v /opt/tgi-cache:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id mistralai/Mistral-7B-Instruct-v0.2
```
Should get you up and running with a local instance of Mistral 7B Instruct on the localhost at port 8080, that can be tested with
```bash
curl http://localhost:8080/ \
-X POST \
-H 'Content-Type: application/json' \
-d '{"inputs":" [INST] What was the latest released hero for Dota 2? [/INST] The latest released hero for Dota 2 was", "max_tokens": 200}'
```
### Running with Ollama
[Ollama](https://ollama.com) is another popular option for running open source models locally. It simplifies model management and provides an OpenAI-compatible API.
```plaintext
ollama pull mistral
ollama serve
```
This starts a local server at `http://localhost:11434` that Discourse AI can connect to using the **Ollama** provider.
### Making it available for your Discourse instance
Most of the times you will be running this in a dedicated server because of the GPU requirement. When doing so I recommend running a reverse proxy, doing TLS termination and securing the endpoint so it can only be connected by your Discourse instance.
### Configuring Discourse AI
LLM connections are now configured through the admin UI rather than site settings. Navigate to `/admin/plugins/discourse-ai/ai-llms` and add a new LLM:
1. Click **New** to add a model
2. Select a **Provider** — choose **vLLM** , **Hugging Face** , or **Ollama** depending on your inference server
3. Enter the **URL** of your inference endpoint (e.g. `http://your-server:8080`)
4. Enter an **API key** if your endpoint requires one
5. Fill in the **model name** , **tokenizer** , **max prompt tokens** , and other model details
Once your LLM is added, set it as the default via the **ai\_default\_llm\_model** site setting, or assign it to specific features through their agent configuration in `/admin/plugins/discourse-ai/ai-features`.
> Last edited by @JammyDodger 2024-05-25T11:01:36Z
>
> > **Check document**
> >
> > Perform check on document:
---
### Author:  [@Bathinda](https://meta.discourse.org/u/Bathinda)
#### Post date: [March 19, 2024, 4:44am UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/3 "2024-03-19T04:44:43Z")
For anyone searching this topic with/for:
#Llava-Api-keys
---
### Author:  [@Isambard](https://meta.discourse.org/u/Isambard)
#### Post date: [March 23, 2024, 10:48pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/4 "2024-03-23T22:48:26Z")
I’m using vLLM too. I also would recommend the openchat v3.5 0106 model, which is a 7B parameter model which performs very well.
I actually am running it in 4bit quantized so that it runs faster.
---
### Author:  [@oppman](https://meta.discourse.org/u/oppman)
#### Post date: [January 13, 2025, 11:43pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/5 "2025-01-13T23:43:35Z")
> [@Discourse](#):
>
> dedicated server because of the GPU requirement. When doing so I recommend running a reverse proxy, doing TLS termination and securing the endpoint so it can only be connected by your Discourse instance.
I am assigning this task to an intern. Are there recommendations from anyone on what specific service to sign up for? This is for a test. The intern currently has a test configured with OpenAI. It runs fine. They’re interested in trying the HuggingFace TGI, but it seems that I need to give them a dedicated server with GPU? What’s the minimal specs for a test?
Are there links I can give the intern?
I haven’t looked at this project in depth yet. I am just anticipating that the intern will need some resources and I am trying to make some reasonable recommendations on services for the intern in research.
---
### Author:  [@Eric\_Keller](https://meta.discourse.org/u/Eric_Keller)
#### Post date: [January 15, 2025, 4:16pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/6 "2025-01-15T16:16:40Z")
Hey there, while exposing with a self-signed certificate vllm container on a GPU box on-prem, I did not find a good way to add the Root CA to the discourse container so it can access securely this on-prem service over https.
e.g.:
```plaintext
./launcher enter app
curl -L https://vllm.infra.example.com/v1/models
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```
Is there a good way to add a self-signed root ca-certificate in the discourse container which would survive container image updates?
As far as I know adding it in the app.yml
```plaintext
run:
- exec: wget ... && update-ca-certificates
```
would only work well while building/rebuilding the app.
Any hint welcomed.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [February 21, 2025, 2:37pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/7 "2025-02-21T14:37:56Z")
14 posts were split to a new topic: [Getting discourse ai to work with ollama locally](https://meta.discourse.org/t/getting-discourse-ai-to-work-with-ollama-locally/353614)
---
### Author:  [@pacharanero](https://meta.discourse.org/u/pacharanero)
#### Post date: [January 20, 2026, 5:17pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/8 "2026-01-20T17:17:35Z")
Would there be any way to add a custom HTTP header to the Discourse AI requests? We have a GPU-equipped server which we use for a variety of purposes, it has an OpenAI-compatible API but for security we have put it behind an API Management gateway, which requires a custom HTTP header (something like `SubscriptionKey`) and we supply our auth key here.
Note that this auth key is to get through the APIM, and is different from the API key one would supply for a service like the OpenAI API.
It would be great if there was a way to add Custom Headers in the setup of each LLM in the `admin/plugins/discourse-ai/ai-llms` setup page. Perhaps a 'Custom option in the ‘Provider’ dropdown, with additional options for use-cases like this?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [January 20, 2026, 7:26pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/9 "2026-01-20T19:26:40Z")
The quick answer is a plugin to add that header. A work-around might be a local proxy server that you’d hit from Discourse and it’d add the header and ferry the data back to Discourse. An untrusted source suggests that you might be able to get NGINX to do that for you with something like
```plaintext
server {
listen 80;
location / {
proxy_pass https://api.example.com;
proxy_set_header SubscriptionKey xyz;
proxy_set_header Host $host;
}
}
```
There could be a clever way to add that to Discourse’s nginx config at some secret path (that also listened only to your server)
---
### Author:  [@mononym](https://meta.discourse.org/u/mononym)
#### Post date: [July 6, 2026, 4:34pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/10 "2026-07-06T16:34:04Z")
Hello,
I can’t find these settings:
> [@Discourse](#):
>
> ### Configuring Discourse AI
>
> LLM connections are now configured through the admin UI rather than site settings. Navigate to `/admin/plugins/discourse-ai/ai-llms` and add a new LLM:
>
> 1. Click **New** to add a model
> 2. Select a **Provider** — choose **vLLM** , **Hugging Face** , or **Ollama** depending on your inference server
Any suggestion for trying out the AI translation feature with the help of an Ollama backend ?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [July 6, 2026, 4:54pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/11 "2026-07-06T16:54:41Z")
> [@mononym](#):
>
> Any suggestion for trying out the AI translation feature with the help of an Ollama backend ?
Create a new LLM at `/admin/plugins/discourse-ai/ai-llms/new?llmTemplate=none`, select OpenAI as the provider as Ollama has [OpenAI compatibility - Ollama](https://docs.ollama.com/api/openai-compatibility).
---
### Author:  [@mononym](https://meta.discourse.org/u/mononym)
#### Post date: [July 6, 2026, 5:04pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/12 "2026-07-06T17:04:42Z")
Thanks ! I noticed as well that the URL only works when the AI plugin is enabled 🤦
For the newly added provider, any suggestions what to choose/set for the following required settings ?
> 1. Tokenizer
> 2. Context window
For the model, I was thinking to maybe use [stablelm2:1.6b](https://ollama.com/library/stablelm2:1.6b) or [translategemma:4b](https://ollama.com/library/translategemma:4b) but dunno yet.
In the README of the latter there is some description of an input prompt. ~~I wonder what the query is that the translation AI feature sends to Ollama.~~ c.f. `/admin/plugins/discourse-ai/ai-agents/-28/edit`
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [July 6, 2026, 5:12pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/13 "2026-07-06T17:12:01Z")
> [@mononym](#):
>
> Thanks ! I noticed as well that the URL only works when the AI plugin is enabled 🤦
😅
> [@mononym](#):
>
> 1. Tokenizer
The closest to the model you will pick. For Gemma, you can pick the Gemini one.
> [@mononym](#):
>
> 1. Context window
Here you set the max context of your deployment.
> [@mononym](#):
>
> I was thinking to maybe use [stablelm2:1.6b](https://ollama.com/library/stablelm2:1.6b)
Please don’t. That model is very very very very very old. It’s the worst choice you can make.
> [@mononym](#):
>
> [translategemma:4b](https://ollama.com/library/translategemma:4b)
That model is one generation behind, and it has very strict prompt requirements that aren’t compatible with Discourse. We expect a more standard general use LLM.
At those sizes, I recommend either Qwen 3.5 or Gemma 4.
---
### Author:  [@mononym](https://meta.discourse.org/u/mononym)
#### Post date: [July 6, 2026, 6:12pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/14 "2026-07-06T18:12:45Z")
Thanks again. I put the settings, pulled the model, tweaked the agent and started the AI translation feature now. Let’s see what’s coming.
Another question. We were using the translator plugin before with a generously provided LibreTranslate instance. Is it not possible to plug the Discourse AI translation feature into that LibreTranslate endpoint ? In the end, that would be much easier than having to host an Ollama instance on a server without true LLM capabilities.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [July 6, 2026, 6:36pm UTC](https://meta.discourse.org/t/self-hosting-an-opensource-llm-for-discourseai/290923/15 "2026-07-06T18:36:54Z")
> [@mononym](#):
>
> Another question. We were using the translator plugin before with a generously provided LibreTranslate instance.
Why not stay using that then? We are not updating it anymore, but it will keep working as-is.
> [@mononym](#):
>
> Is it not possible to plug the Discourse AI translation feature into that LibreTranslate endpoint ? In the end, that would be much easier than having to host an Ollama instance on a server without true LLM capabilities.
The Discourse AI translation operates with a whole different set of constraints than the original Translator plugin, so we built it from the ground up to address this new world, where LLM based translations are better and cheaper that what was possible before with ML translation APIs.