# How to add a new Chat Bot connected to a self-hosted LLM?

**URL:** <https://meta.discourse.org/t/how-to-add-a-new-chat-bot-connected-to-a-self-hosted-llm/299791>\
**Category:** Development\
**Tags:** ai, ai-bot\
**Created:** [March 6, 2024, 11:18pm UTC](https://meta.discourse.org/t/how-to-add-a-new-chat-bot-connected-to-a-self-hosted-llm/299791 "2024-03-06T23:18:40Z")\
**Posts on this page:** 1\
**Showing post:** 10

<div class="post-metadata">

**Author:** ![Isambard](https://avatars.discourse-cdn.com/v4/letter/i/858c86/32.png) [@Isambard](https://meta.discourse.org/u/Isambard)\
**Post date:** [June 4, 2024, 7:42pm UTC](https://meta.discourse.org/t/how-to-add-a-new-chat-bot-connected-to-a-self-hosted-llm/299791/10 "2024-06-04T19:42:09Z")

</div>

I’m running several different things. For Discourse stuff, I will run a 7B model based off Mistral and fine-tuned for the tasks. I’m looking at various BERT-like models for classification tasks and still undecided on the embeddings yet. This runs on a 2nd hand 3090 Ti which I bought for $700.

I would love to have an A100, but instead, I built a separate 4 GPU system ‘on the cheap’ for only $1,000 that runs Llama 3 70Bq4 at over 20 tok/s.

For sure in many/most cases it would make sense to just go with a provider, however, it might make sense to DIY if:

- You want to learn
- You want to have control certainty over your models (so you don’t lose access to them, or are beholden to a company to use their non-public embeddings)
- You have a lot of bulk processing to do which would be cheaper to do in-house
- You want reserved and reliable capacity (there are limits on both requests and tokens available from providers) for bulk processing

---

_[View the full topic](https://meta.discourse.org/t/how-to-add-a-new-chat-bot-connected-to-a-self-hosted-llm/299791)._