Support for Mistral API

Contribute Feature

AI-generated summary

Crashley has been experimenting with Mistral’s new models (mistral-small and mistral-medium) via their API and finds mistral-small — a Mixtral 8x7B MOE model — competitive with GPT-3.5 and potentially a cost-effective alternative to GPT-4 for certain use cases. They request support for the Mistral API in the Discourse AI plugin, noting that Mistral’s API follows OpenAI’s format, suggesting a path for generic vendor support. Crashley proposes refactoring AI settings to accept generic endpoints (chat, models, embeddings) and API keys, allowing Discourse to dynamically fetch model names or accept manual entry.

sam confirms that Mixtral is now available via vLLM and that Roman is working on a “generic LLM” interface to better support diverse models. They agree that the current AI settings are too limited and advocate for a new, scalable interface that lists available LLMs, explains their endpoints, parameters, quotas, and more — necessary given the rapidly expanding landscape of models. The consensus is that Discourse needs a more flexible, future-proof system for integrating LLMs beyond hardcoded vendor support.

1

I’ve been playing with the new Mistral models (mistral-small and mistral-medium) via their API and like what I’m seeing. Mistral-small (which is Mixtral 8x7B MOE), appears to outperform GPT3.5 and could be a viable alternative (even if not quite as capable) to GPT4 for some use cases. Most importantly, it’s significantly cheaper than GPT4. Would love to see support for the Mistral API in the Discourse AI plugin.

I tried dropping the Mistral API URL and token into the openAI GPT4 turbo fields, but not surprisingly that didn’t work because it was requesting the wrong model. Looking at the API documentation for openAI, Anthropic and Mistral, they all use the same format. I imagine this is on purpose - new vendors aligning to openAI so that they can be used as drop-in replacements.

So part 2 of this feature request is to consider refactoring the AI settings to be more generic to accommodate any vendor that adopts an openAI-like interface. In the settings, simply provide 4 things: the chat endpoint, the model list endpoint, the embeddings endpoint, and the API key. Discourse then queries the /models endpoint to fetch the names, and/or you can type in model names manually.

2

@Falco just landed mixtral today via vLLM.

@Roman is working on refactoring our internal implementation so bot can lean on our new “generic LLM” interface.

I agree about rethinking all AI settings when it comes to LLMs, the patterns we need outgrow what you can do in site settings.

A new interface listing all LLMs you have access to, explaining endpoints / params / quotas and more is very much something we need to start thinking about. The current way is just a bit too limited and given there are infinity models out there, we need a new paradigm here.

3 Likes