Just to expand here a bit:
https://chat.lmsys.org/?leaderboard
Mistral comes in many flavors … there is Mistral 7b, Mixtral 8x7b (the one you have), and the brand new mistralai/Mixtral-8x22B-Instruct-v0.1 · Hugging Face - this and another 5/6 models they release including some closed source ones.
Got to be careful with a “Mistral not good enough” and always clarify
I would say Mixtral-8x7b is simply not a great fit for tool support, it strays off too much.
I would say it is
- Pretty good for “upload” support
- Very good at custom persona support
- Weak at tool support
We are trying to see if we can upgrade to 8x22b (it ships with good tool support), trouble is that memory requirements are quite high and we would need to quantize the model to fit it nicely on our servers.
But really… if you have a data privacy deal with Amazon I would strongly recommend bedrock which would give you access to Claude 3 Opus and Haiku.
I do get the tension between open source models vs closed source ones. Its tough the closed source ones are just quite a bit ahead at the moment.