How can I restrict an AI Agent to answer only questions related to my forum?

Support
,
1

I’m setting up Discourse AI on my forum and have an AI Bot/Agent working with Gemini 2.5 Flash via LiteLLM and Google Vertex AI.

The bot is working, but I have a problem with its scope.

For example, my forum is a business-focused community. If I ask the bot:

“Who owns Uber?”

it answers the question using the LLM’s general knowledge, even though the question has nothing to do with my forum.

This creates two problems:

  1. I want the bot to function as a forum-specific knowledge assistant, not as a general-purpose Gemini/ChatGPT interface.
  2. I don’t want users to be able to ask arbitrary questions and consume my LLM credits.

What I would ideally like is:

User asks question
↓
Is it relevant to this forum?
↓
NO → politely refuse
↓
YES
↓
Search forum knowledge
↓
Generate answer based on forum content

image
image835×808 65.5 KB

2

Have you taken a look at Agents? Perhaps the Forum Researcher or Forum Helper Agents would be a better fit, as they are tuned to work with the forum specifically. Or is that what you are already using?

3

Forum Helper also has this problem:

image

4

Hmm… then perhaps you might want to make your own agent with a customized system prompt to not respond to queries outside of your forum’s scope.

5

Here’s what I see today if I try this on ask.discourse.com

Thanks for the question! Since you asked on a Discourse support channel, I’ll answer in the context of how language models power Discourse’s AI features — but let me know if you actually meant a general technical explanation of how LLMs work, and I can point you elsewhere for that.

Someone round here must know how that catch was made.

6

The “Creative” agent you used in the test is our “no system prompt” agent, that just passes the request to the underlying LLM.

You will want to craft a new agent, with a system prompt that specifies how and when to answer on your own forum.

That’s what we did on ask.discourse.com as mentioned by @Ed_S.

7

Although prompts can influence an agent’s behavior, they aren’t always reliable. I was able to trick the agent at ask.discourse.com into answering a few off-topic questions.

image
image970×960 49.7 KB

8

(This is probably one of those situations where 100% enforcement would cause more inconvenience than benefit. )

9

Yeah, after a conversation gets deep and you chain it, you can get it to keep on going. Not a worry for us as we host the models ourselves, so we don’t pay per token anyway. We also analyze all conversations on Ask daily and can ban any bad actors. The forum moderation flow is still the same.

If you want even more control than what the system prompt gives you, you can try some approaches:

Workflows

Our new workflow plugin would allow you to do a 2-pass on every message, first with an agent evaluating if a message should be replied to, then another conditionally replying.

Fine-tuned models

The most surefire way to control is to deploy a model that is actually trained with the behavior you want.

This is easy with Discourse AI, as we allow you to pick and choose LLMs for each agent.