I’m new to discourse AI. I’m using “sentence-transformers/all-mpnet-base-v2” as my embedding model. is this enough to do semantic search?
Or Should i have to add a Hyde model for it?
Please guide me on this.
You also need an LLM for semantic search. If you want to self-host see Self-Hosting an OpenSource LLM for DiscourseAI.
Thank you so much.
Can you gimme an idea on the requirements to host model like “mistralai/Mistral-7B-Instruct-v0.2” on-prem and in cloud as well for an enterprise level website, please.
And also i cant able to find any tokenizers for this model in the admin panel.
There is nothing Discourse specific here, so standard rules apply. A 7B model, if ran using fp16, will take ~14GB VRAM plus the space for the context. You can use fp8 quantization to halve that, but that old model isn’t the best for it.
As it isn’t feasible to ship every possible tokenizer, you should pick the closest one from the available tokenizers.
1 Like
Can you suggest us some LLM models for this scenario? We might be hosting our model on prem, so would like to know the compatibility factor of the models with Discourse.
Thank you.
Depends on you budget, language support target, and what features of Discourse AI you want.
Today Qwen 2.5 Instruct in 32B or 72B are a strong contender.
Is there any way that we can use a smaller model for the summarization feature? Because LLM’s take a bigger budget, and we might have to settle for something smaller for now…
Yes, you can use any model you want.
1 Like