Is there a way to Add an API key from an LLM that is not from those listed?
We use this LLM called Deepseek and would like to integrate it. Already have the Api keys etc.
1 Like
AFAIK, no, as Discourse probably only supports those listed.
If I am wrong, anyone, feel free to contradict me.
1 Like
It works with VLLM, you can can just configure a VLLM endpoint and it should work out of the box.
5 Likes
WOW !!
Thanks so much. I will try this out.
2 Likes
Having support for almost every modern LLM that exist was a big focus for release 3.3, and we finally got to it, just in time when open weights LLMs are catching up to closed ones.
Let me know if you need help configuring DeepSeek in your instance, this feature is brand new and we are still updating all the documentation to reflect it’s new capabilities.
1 Like
Yes I do need help. Looked around but don’t see how to add it without VLLM. VLLM is another overhead I would rather avoid due the requirements.
Would be great if there’s an integration not only for Deepseek, but for many other Open source LLMs. I am of the opinion that’s where this LLM race is heading.
From what we see, companies are hesitant handing out their Data to the “cloud”. Open source that allows them to run local instances using tools like GPT4ALL, HuggingFace will end up being the preferred model.
Thanks
Yes, I agree with that, which is why we support running your own LLM from day one.
What?
vLLM is the most popular inference engine, and it’s far from being an “overhead”, as deploying with it means you will get faster inference than any cobbled together python script a developer can write in a couple of days and it’s a simple docker run --gpus all vllm/vllm-openai:latest --model deepseek-ai/DeepSeek-Coder-V2-Instruct away for most models.
Our implementation will work with any API that adheres to the usual OpenAI defacto spec, so you can run it without vLLM or Hugging Face TGI as long as you make your API adhere to what is expected.
Thanks, from my reading, {may have read it wrong) don’t I need a new Droplet [Digital Ocean] aka, VPS to deploy a VLLM? or can it be deployed on the same Droplet that hosts my Discourse Instance?
If yes, on the same instance, would it be on the same directory as Discourse or a separate Directory?
Thanks again
What exact droplet are you running your current Discourse site? The $6 one ?
The $15 one. I think it is 2 GB ram. Once we finish testing I will scale it up.
Thanks
Hmm, are you aware that a droplet to run this Deepseek model you want will cost you like $10,000 a month?
3 Likes
I guess @oga wants to use the official Deepseek API service instead of deploying an open-source model on their own.
Deepseek’s official API is compatible with OpenAI’s API, so just need to add a new LLM under admin/plugins/discourse-ai/ai-llms.
2 Likes
PRECISELY what we are trying to do. We are setting up a platform to train Young kids and folks on how to write effective PROMPTS, but we want to do it on an environment that we can monitor what they Prompt for.
Let me try this and see. Deploying and Open-Source model for this will be cost prohibitive.
Thanks
1 Like
Holy Chicken Nuggets !! Not what we want nor can afford.
1 Like