@Falco just landed mixtral today via vLLM.
@Roman is working on refactoring our internal implementation so bot can lean on our new “generic LLM” interface.
I agree about rethinking all AI settings when it comes to LLMs, the patterns we need outgrow what you can do in site settings.
A new interface listing all LLMs you have access to, explaining endpoints / params / quotas and more is very much something we need to start thinking about. The current way is just a bit too limited and given there are infinity models out there, we need a new paradigm here.