Discourse AI - Related topics

[Discourse]

This guide explains how to enable and configure the Related topics feature of the Discourse AI plugin.

Required user level: Administrator

Related topics helps users discover relevant content by suggesting semantically similar topics based on the one they’re currently reading. This enhances content exploration and increases user engagement.

Features

• Semantic textual similarity: Goes beyond keyword matching to find truly related content
• Toggle between “Suggested” and “Related” topics
• Available for both anonymous and logged-in users

Enabling Related topics

Related topics is turned on by default for all Discourse hosted customers with the Discourse AI plugin enabled

Prerequisites

Related topics requires Embeddings to function.

If you are on our hosting, Embeddings is provided using an open-source model. No additional setup is required.

Self-hosted instances will need to configure an embedding model through a supported provider.

Configuration

1. Go to Admin → Plugins → Discourse AI → AI Features
2. Find the Embeddings module and configure it:
   • Set ai_embeddings_selected_model to an embedding definition you have configured
   • Enable ai_embeddings_enabled to activate Embeddings
3. Enable ai_embeddings_semantic_related_topics_enabled to activate the Related Topics feature

Setting up an embedding model

Before enabling embeddings, you need to configure an embedding model. Go to Admin → Plugins → Discourse AI → Embeddings to create a new embedding definition. You can choose from several presets:

• Open AI: text-embedding-3-small or text-embedding-3-large (recommended for most sites)
• Google: gemini-embedding-001
• Hugging Face (self-hosted inference): multilingual-e5-large (recommended for non-English or multilingual sites), bge-large-en, or bge-m3

You will need to provide an API key (or link an AI Secret) and endpoint URL for your chosen provider.

Additional settings

The following settings let you fine-tune the Related Topics feature:

• ai_embeddings_semantic_related_topics: Maximum number of topics to show in the related topics section (default: 5)
• ai_embeddings_semantic_related_include_closed_topics: Whether to include closed topics in related results (default: true)
• ai_embeddings_semantic_related_age_penalty: Apply a penalty to older topics so newer content is preferred (default: 0.0, range: 0.0–2.0)
• ai_embeddings_semantic_related_age_time_scale: Time scale in days for the age penalty (default: 365)

Technical FAQ

Expand to view a diagram of the Related topics architecture

The overview is, that when a topic is created / updated this happens:

sequenceDiagram
    User->>Discourse: Creates topic
    Discourse-->>Embedding Microservice: Generates embeddings
    Embedding Microservice-->>Discourse: 
    Discourse-->>PostgreSQL:Store Embeddings 

And during topic visit:

sequenceDiagram
    User->>Discourse: Visits topic
    Discourse-->>PostgreSQL: Query closest topics
    PostgreSQL-->>Discourse: 
    Discourse->>User: Presents related topics 

How does Related topics work?

• When a user visits a topic, Discourse queries the database for the most semantically similar topics based on their embedded representations. These related topics are then presented to the user, encouraging further exploration of the community’s content.

How is topic/post data processed?

• For Discourse-hosted sites, data is processed within our secure virtual private datacenter. For self-hosted sites, data processing depends on your chosen third-party provider.

Where is the embeddings data stored?

• Embeddings data is stored in your Discourse database, alongside other forum data like topics, posts, and users.

What embedding models are available?

• Discourse AI supports models from OpenAI (text-embedding-3-small, text-embedding-3-large), Google (gemini-embedding-001), Hugging Face-compatible endpoints (bge-large-en, bge-m3, multilingual-e5-large), and Cloudflare Workers AI. You can also configure custom embedding models through the admin UI.

Last edited by @Saif 2024-11-04T18:08:05Z

Last checked by @hugh 2024-08-06T04:30:59Z

Check document

Perform check on document: