I want to add a new “Chat Bot” and link it to a self hosted LLM.
I have tried to use the “ai hugging face model display name” field and that doesn’t seem to appear anywhere, perhaps I have to reference that in the prompts associated with a persona?
I have also tried to “create” a new bot via the “ai bot enable chat bots” drop down, and anything I create appear shows in the chatbot drop down as " [en.discourse_ai.ai_bot.bot_names.XXXX] where XXXX is the name I provided.
Any tips to any documentation or guide who to do this would be appreacaited.
AI-generated summary
In a Dev discussion titled “How to add a new Chat Bot connected to a self-hosted LLM?”, Nadeem seeks guidance on linking a self-hosted LLM to a Discourse chatbot, noting that the “ai hugging face model display name” field doesn’t appear to work and newly created bots show up with placeholder names like [en.discourse_ai.ai_bot.bot_names.XXXX]. sam confirms that this functionality is currently unavailable but is under active refactoring by Roman, with improvements expected in 1–2 weeks. Isambard expresses surprise, noting that since OpenAI is supported, self-hosted LLMs with OpenAI-compatible endpoints should logically work too. Isambard shares cost estimates for self-hosting: minimal incremental electricity cost (~$5/month at idle for a GPU already in use), and notes that for small forums, using an LLM-as-a-service may be more economical, but for larger-scale Discourse deployments, internal hosting could be both cost-effective and strategically valuable. He runs a fine-tuned 7B Mistral-based model on a secondhand RTX 3090 Ti ($700), and a 4-GPU system for $1,000 capable of running Llama 3 70Bq4 at 20+ tok/s. He benchmarks his 3090 at 2600 tok/s for Llama 3 8B FP16, costing ~$0.007–$0.01 per million tokens including depreciation — competitive with Claude Haiku at scale. Isambard also notes that even CPU-only web servers can run small LLMs at modest speeds (~6 tok/s), useful for background tasks. sam asks about A100 pricing and models used, to which Isambard replies he’s using Mistral-based and BERT-like models for classification, and is still evaluating embeddings.
Anyone who can offer any suggestions or is this a known limitation?
@Roman is working on refactoring this section, expect more news in the coming weeks
I’m not sure if I interpret this correctly that currently it is not possible to use a self-hosted LLM, but this will change soon?
It is not possible atm, but hopefully in a week or 2 we will have this working.
Thanks. I was surprised it didn’t work since OpenAI is supported. I think many people run their own LLMs with an OpenAI compatible endpoint. I will look forward to the update in 2 weeks 
Out of interest @Isambard what’s your estimate for how much a sufficiently powerful local LLM will cost you to host on a monthly basis (dollar equivalent)?
About a minimum of $5 in additional electricity costs per month for the GPU at idle - although in reality, the incremental cost for discourse is zero since I already run the LLM for other purposes.
But for sure, it would be more economical for small forums and low usage to use an LLM as a service. Though for the scale of Discourse’s hosted offering, I suspect it might make sense to host internally (and also develop knowledge of this area that is likely to be important).
And 15k for the A100 ?
What model particularly are you running locally?
I’m running several different things. For Discourse stuff, I will run a 7B model based off Mistral and fine-tuned for the tasks. I’m looking at various BERT-like models for classification tasks and still undecided on the embeddings yet. This runs on a 2nd hand 3090 Ti which I bought for $700.
I would love to have an A100, but instead, I built a separate 4 GPU system ‘on the cheap’ for only $1,000 that runs Llama 3 70Bq4 at over 20 tok/s.
For sure in many/most cases it would make sense to just go with a provider, however, it might make sense to DIY if:
- You want to learn
- You want to have control certainty over your models (so you don’t lose access to them, or are beholden to a company to use their non-public embeddings)
- You have a lot of bulk processing to do which would be cheaper to do in-house
- You want reserved and reliable capacity (there are limits on both requests and tokens available from providers) for bulk processing
I benchmarked the 3090 and was getting max sustained throughput of around 2600 tokens per second running Llama 3 - 8B FP16. I live in an expensive electricity region but running continuously at a 285W power limit, it would cost around $0.007 per million output tokens. Or roughly $0.01 per million tokens if you fully depreciate the equipment cost over 3 years.
This compares quite favourably to Claude Haiku providing you have a reasonable utilization rate.
I made an interesting discovery: the web server that I’m hosting my forum on has sufficient grunt to run a small LLM at modest speeds (6 tok/s without batching) even without a GPU. This will be useful for offline/background tasks.