I’m new to discourse AI. I’m using “sentence-transformers/all-mpnet-base-v2” as my embedding model. is this enough to do semantic search? Or Should i have to add a Hyde model for it? Please guide me on this.

[image] Ajindiscource: Or Should i have to add a Hyde model for it? You also need an LLM for semantic search. If you want to self-host see Self-Hosting an OpenSource LLM for DiscourseAI.

Thank you so much. Can you gimme an idea on the requirements to host model like “mistralai/Mistral-7B-Instruct-v0.2” on-prem and in cloud as well for an enterprise level website, please. And also i cant able to find any tokenizers for this model in the admin panel.

[image] Ajindiscource: Can you gimme an idea on the requirements to host model like “mistralai/Mistral-7B-Instruct-v0.2” on-prem and in cloud as well for an enterprise level website, please. There is nothing Discourse specific here, so standard rules apply. A 7B model, if ran using fp16, will take ~14GB VRAM plus the space for the context. You can use fp8 quantization to halve that, but that old model isn’t the best for it. [image] Ajindiscource: And also i cant able to find any tokenizers for this model in the admin panel. As it isn’t feasible to ship every possible tokenizer, you should pick the closest one from the available tokenizers.

Can you suggest us some LLM models for this scenario? We might be hosting our model on prem, so would like to know the compatibility factor of the models with Discourse. Thank you.

Depends on you budget, language support target, and what features of Discourse AI you want. Today Qwen 2.5 Instruct in 32B or 72B are a strong contender.

Yes, you can use any model you want.

Can we alter the payload to the model for summarization or any other function? Especially, I want to change the system’s content. {‘role’: ‘system’, ‘content’: ‘You are an advanced summarization bot that generates concise, coherent summaries of provided text.\n\n- Only include the summary, without any additional commentary.\n- You understand and generate Discourse forum Markdown; including links, italics, bold.\n- Maintain the original language of the text being summarized.\n- Aim for summaries to be 400 words or less.\n- Each post is formatted as “ ) ”\n- Cite specific noteworthy posts using the format DESCRIPTION\n - Example: links to the 3rd and 6th posts by sam: sam (#3, #6)\n - Example: link to the 6th post by jane: agreed with\n - Example: link to the 13th post by joe: joe\n- When formatting usernames either use @USERNMAE OR USERNAME’} If possible, how to do it?

That is not possible at the moment, but we already started work on making it possible. Should land in the coming weeks.

Adding Semantic Search feature for our self-hosted discourse site

Product Support

Abinav_22 (Zecos22) March 10, 2025, 5:19am 7

Is there any way that we can use a smaller model for the summarization feature? Because LLM’s take a bigger budget, and we might have to settle for something smaller for now…

Topic		Replies	Views
Setup Inquiry: AI Summarization in Discourse & LLM Integration Product Support ai	10	542	October 6, 2025
Discourse AI - Self-Hosted Guide Self-Hosting ai	59	15344	May 20, 2024
What LLM to use for Discourse AI? Site Management how-to , ai	0	1115	January 23, 2025
Estimating cost of enabling Discourse AI for related content and search Product Support ai	2	324	October 28, 2025
Discourse AI - Large Language Model (LLM) settings page Site Management how-to , ai	1	4869	September 30, 2024

Adding Semantic Search feature for our self-hosted discourse site

Related topics