How can I restrict an AI Agent to answer only questions related to my forum?

Support
,
1

I’m setting up Discourse AI on my forum and have an AI Bot/Agent working with Gemini 2.5 Flash via LiteLLM and Google Vertex AI.

The bot is working, but I have a problem with its scope.

For example, my forum is a business-focused community. If I ask the bot:

“Who owns Uber?”

it answers the question using the LLM’s general knowledge, even though the question has nothing to do with my forum.

This creates two problems:

  1. I want the bot to function as a forum-specific knowledge assistant, not as a general-purpose Gemini/ChatGPT interface.
  2. I don’t want users to be able to ask arbitrary questions and consume my LLM credits.

What I would ideally like is:

User asks question
↓
Is it relevant to this forum?
↓
NO → politely refuse
↓
YES
↓
Search forum knowledge
↓
Generate answer based on forum content

image
image835×808 65.5 KB

2

Have you taken a look at Agents? Perhaps the Forum Researcher or Forum Helper Agents would be a better fit, as they are tuned to work with the forum specifically. Or is that what you are already using?

3

Forum Helper also has this problem:

image

4

Hmm… then perhaps you might want to make your own agent with a customized system prompt to not respond to queries outside of your forum’s scope.

5

Here’s what I see today if I try this on ask.discourse.com

Thanks for the question! Since you asked on a Discourse support channel, I’ll answer in the context of how language models power Discourse’s AI features — but let me know if you actually meant a general technical explanation of how LLMs work, and I can point you elsewhere for that.

Someone round here must know how that catch was made.

6

The “Creative” agent you used in the test is our “no system prompt” agent, that just passes the request to the underlying LLM.

You will want to craft a new agent, with a system prompt that specifies how and when to answer on your own forum.

That’s what we did on ask.discourse.com as mentioned by @Ed_S.