I want to add a new “Chat Bot” and link it to a self hosted LLM. I have tried to use the “ai hugging face model display name” field and that doesn’t seem to appear anywhere, perhaps I have to reference that in the prompts associated with a persona? I have also tried to “create” a new bot via the “…

Anyone who can offer any suggestions or is this a known limitation?

@Roman is working on refactoring this section, expect more news in the coming weeks

I’m not sure if I interpret this correctly that currently it is not possible to use a self-hosted LLM, but this will change soon?

Thanks. I was surprised it didn’t work since OpenAI is supported. I think many people run their own LLMs with an OpenAI compatible endpoint. I will look forward to the update in 2 weeks :slight_smile:

Out of interest @Isambard what’s your estimate for how much a sufficiently powerful local LLM will cost you to host on a monthly basis (dollar equivalent)?

About a minimum of $5 in additional electricity costs per month for the GPU at idle - although in reality, the incremental cost for discourse is zero since I already run the LLM for other purposes. But for sure, it would be more economical for small forums and low usage to use an LLM as a service. …

[תמונה] Isambard: for the GPU at idle And 15k for the A100 ? What model particularly are you running locally?

I’m running several different things. For Discourse stuff, I will run a 7B model based off Mistral and fine-tuned for the tasks. I’m looking at various BERT-like models for classification tasks and still undecided on the embeddings yet. This runs on a 2nd hand 3090 Ti which I bought for $700. I wou…

I benchmarked the 3090 and was getting max sustained throughput of around 2600 tokens per second running Llama 3 - 8B FP16. I live in an expensive electricity region but running continuously at a 285W power limit, it would cost around $0.007 per million output tokens. Or roughly $0.01 per million to…

איך להוסיף בוט צ'אט חדש המחובר ל־LLM שמארחים בעצמכם

מפתח

sam (Sam Saffron) 3 ביוני,‏ 2024,‏ 2:11am 5

It is not possible atm, but hopefully in a week or 2 we will have this working.

נושא		תגובות	צפיות
How to configure Discourse to use a locally installed LLM? Support ai	7	479	3 ביוני,‏ 2025
Estimating cost of enabling Discourse AI for related content and search Support ai	2	255	28 באוקטובר,‏ 2025
Adding a new Chat Bot when using AI plugin Support ai	0	467	12 במרץ,‏ 2024
Self-Hosting an OpenSource LLM for DiscourseAI Self-Hosting ai	12	4263	6 ביולי,‏ 2026
How to use the hugging face llama2 chat bot Development ai , ai-bot	2	685	9 במרץ,‏ 2024

איך להוסיף בוט צ'אט חדש המחובר ל־LLM שמארחים בעצמכם

נושאים קשורים