Can I add other LLM?

[Falco]

@oga

From what we see, companies are hesitant handing out their Data to the “cloud”. Open source that allows them to run local instances using tools like GPT4ALL, HuggingFace will end up being the preferred model.

Yes, I agree with that, which is why we support running your own LLM from day one.

@oga

VLLM is another overhead I would rather avoid due the requirements.

What?

vLLM is the most popular inference engine, and it’s far from being an “overhead”, as deploying with it means you will get faster inference than any cobbled together python script a developer can write in a couple of days and it’s a simple docker run --gpus all vllm/vllm-openai:latest --model deepseek-ai/DeepSeek-Coder-V2-Instruct away for most models.

@oga

Looked around but don’t see how to add it without VLLM.

Our implementation will work with any API that adheres to the usual OpenAI defacto spec, so you can run it without vLLM or Hugging Face TGI as long as you make your API adhere to what is expected.