Self-Hosting an OpenSource LLM for DiscourseAI

[Discourse]

The Discourse AI plugin has many features that require an LLM to be enabled, like, for example, Summarization, AI Helper, AI Search, AI Bot. While you can use a third party API, like Configure API Keys for OpenAI or Configure API Keys for Anthropic we built Discourse AI since first day to not be locked into those.

Running with HuggingFace TGI

HuggingFace provides an awesome container image that can get you running quickly.

For example:

mkdir -p /opt/tgi-cache
docker run --rm --gpus all --shm-size 1g -p 8080:80 \
  -v /opt/tgi-cache:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id mistralai/Mistral-7B-Instruct-v0.2

Should get you up and running with a local instance of Mistral 7B Instruct on the localhost at port 8080, that can be tested with

curl http://localhost:8080/ \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{"inputs":"<s>[INST] What is your favourite condiment? [/INST] Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> [INST] Do you have mayonnaise recipes? [/INST]","parameters":{"max_new_tokens":500, "temperature":0.5,"top_p": 0.9}}'

Running with vLLM

Another option to self-host LLMs Discourse AI supports is vLLM, which is a very popular project, licensed under the Apache License.

Here how to get started with a model:

mkdir -p /opt/vllm-cache
docker run --gpus all \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e "MODEL=mistralai/Mistral-7B-Instruct-v0.2" \
  -p 8080:8000 --ipc=host vllm/vllm-openai:latest

Which you can test with

curl -X POST http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistralai/Mistral-7B-Instruct-v0.2",
"prompt": "<s> [INST] What was the latest released hero for Dota 2? [/INST] The latest released hero for Dota 2 was", "max_tokens": 200}'

Running with Ollama

Ollama is another popular option for running open source models locally. It simplifies model management and provides an OpenAI-compatible API.

ollama pull mistral
ollama serve

This starts a local server at http://localhost:11434 that Discourse AI can connect to using the Ollama provider.

Making it available for your Discourse instance

Most of the times you will be running this in a dedicated server because of the GPU requirement. When doing so I recommend running a reverse proxy, doing TLS termination and securing the endpoint so it can only be connected by your Discourse instance.

Configuring Discourse AI

LLM connections are now configured through the admin UI rather than site settings. Navigate to /admin/plugins/discourse-ai/ai-llms and add a new LLM:

1. Click New to add a model
2. Select a Provider — choose vLLM, Hugging Face, or Ollama depending on your inference server
3. Enter the URL of your inference endpoint (e.g. http://your-server:8080)
4. Enter an API key if your endpoint requires one
5. Fill in the model name, tokenizer, max prompt tokens, and other model details

Once your LLM is added, set it as the default via the ai_default_llm_model site setting, or assign it to specific features through their agent configuration in /admin/plugins/discourse-ai/ai-features.

Last edited by @JammyDodger 2024-05-25T11:01:36Z

Check document

Perform check on document:

[Bathinda (Bathinda Helper)]

For anyone searching this topic with/for:
#Llava-Api-keys

[Isambard]

I’m using vLLM too. I also would recommend the openchat v3.5 0106 model, which is a 7B parameter model which performs very well.

I actually am running it in 4bit quantized so that it runs faster.

[oppman (Craig)]

@Discourse

dedicated server because of the GPU requirement. When doing so I recommend running a reverse proxy, doing TLS termination and securing the endpoint so it can only be connected by your Discourse instance.

I am assigning this task to an intern. Are there recommendations from anyone on what specific service to sign up for? This is for a test. The intern currently has a test configured with OpenAI. It runs fine. They’re interested in trying the HuggingFace TGI, but it seems that I need to give them a dedicated server with GPU? What’s the minimal specs for a test?

Are there links I can give the intern?

I haven’t looked at this project in depth yet. I am just anticipating that the intern will need some resources and I am trying to make some reasonable recommendations on services for the intern in research.

[Eric_Keller]

Hey there, while exposing with a self-signed certificate vllm container on a GPU box on-prem, I did not find a good way to add the Root CA to the discourse container so it can access securely this on-prem service over https.

e.g.:

./launcher enter app
curl -L  https://vllm.infra.example.com/v1/models
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Is there a good way to add a self-signed root ca-certificate in the discourse container which would survive container image updates?

As far as I know adding it in the app.yml

run:
  - exec: wget ... && update-ca-certificates

would only work well while building/rebuilding the app.

Any hint welcomed.

[pacharanero (Marcus Baw)]

Would there be any way to add a custom HTTP header to the Discourse AI requests? We have a GPU-equipped server which we use for a variety of purposes, it has an OpenAI-compatible API but for security we have put it behind an API Management gateway, which requires a custom HTTP header (something like SubscriptionKey) and we supply our auth key here.

Note that this auth key is to get through the APIM, and is different from the API key one would supply for a service like the OpenAI API.

It would be great if there was a way to add Custom Headers in the setup of each LLM in the admin/plugins/discourse-ai/ai-llms setup page. Perhaps a 'Custom option in the ‘Provider’ dropdown, with additional options for use-cases like this?

[pfaffman (Jay Pfaffman)]

The quick answer is a plugin to add that header. A work-around might be a local proxy server that you’d hit from Discourse and it’d add the header and ferry the data back to Discourse. An untrusted source suggests that you might be able to get NGINX to do that for you with something like

server {
    listen 80;
    
    location / {
        proxy_pass https://api.example.com;
        proxy_set_header SubscriptionKey xyz;
        proxy_set_header Host $host;
    }
}

There could be a clever way to add that to Discourse’s nginx config at some secret path (that also listened only to your server)