Experimenting with combined AI search and decision models

News and Events Announcements
,

AI-generated summary

The Discourse team has introduced an experimental “Combined AI search” feature on Meta, which integrates a decision model to automatically determine whether a user’s query is best served by AI or traditional keyword search. As described by awesomerobot, this system uses small, fast models to answer two structured questions: whether the user seeks an explanation (AI) or a specific lookup (keyword), and whether the search should be scoped to the current topic/category or the entire site.

The decision model only acts when confident; otherwise, it relies on built-in fallback rules that consider factors such as search syntax, text writing systems, repeat searches, question phrasing (e.g., trailing ?), and existing tab results. To measure effectiveness, the team tracks metrics including the model’s confidence, which tab opened, and whether users switched away from the AI tab—a key signal for incorrect routing. Repeated switching will adjust learned user preferences.

Site administrators can enable this experimental upcoming change and configure their own decision models via admin > plugins > AI > models using the AI ask AI search intent decision model site setting.

1

We’ve been trying a new header search on Meta that puts AI search and keyword search in the same place, and you decide which one to use.

image
image1144×204 16.3 KB

Today we’re enabling a new step: using a decision model to guess whether you’d be best served by AI or keyword results. It’s available now as the “Combined AI search” experimental upcoming change.

Decision models are small models built to answer structured questions very quickly, with the result being a simple probability. If your site has one set up, combined search asks it two things:

  • Answer or keyword lookup? Does this search want something explained or solved, or a particular topic, page or person?
  • Here or the whole site? For searches made inside a topic or category, is the search about that place or about the forum in general?

It only acts when it’s confident, it’s skipped if it’s slow, and we have some built-in rules if a decision model isn’t available. The rules check:

  • search syntax (operators, #category, @user, quoted phrases)
  • a match inside the topic you’re searching
  • text in different writing systems than the default (so for English as default… Chinese, Arabic, Cyrillic)
  • repeat searches in the same search session
  • your learned preference
  • question phrasing (a trailing ?, five or more words)
  • which tabs actually have results

We’re going to be measuring the effectiveness of this on Meta, so for each search we’ll look at

  • the fallback rules’ verdict
  • the model’s verdict and its confidence
  • which tab opened, and why
  • whether the searcher switched away, and how quickly

Switching away from the AI tab is a good signal that we got it wrong, and if you switch multiple times we’ll remember your preference.

If you want to configure and experiment with this on your own site, you can set up a decision model from admin > plugins > AI > models. When the combined search upcoming change is enabled, you can set your decision model using the AI ask AI search intent decision model site setting.

14 Likes