Discourse AI - Embeddings

This topic covers the configuration of the Embeddings module of the Official Discourse AI Plugin plugin.

:tada: Embeddings is now pre-configured for all customers on our hosting. We provide the embeddings required through open-source models, meaning you can simply enable the AI features that depend on it.

Feature set

The Embeddings modules can automatically generate embeddings of every new post in your Discourse instance. Those are used by our semantic features, like semantic suggested topics and semantic topic search.

When configured, this module will add a “Related Topics” section to the bottom of all topic pages, where topics who are similar to the current page will be linked, helping users find related discussions to what they are currently reading on:

It also adds a Semantic Search option on the full page search:

Providers

This module can use two providers:

  • Open Source: a collection of open source models from SBERT. Recommended and default.

  • OpenAI: Uses OpenAI API to generate embeddings. You will need a working OpenAI API key for this.

Settings

  • ai_embeddings_enabled: Enables or disables the module

  • ai_embeddings_discourse_service_api_endpoint: URL where the API is running for the module. If you are using CDCK hosting this is automatically handled for you. If you are self-hosting check the self-hosting guide.

  • ai_embeddings_discourse_service_api_key: API key for the API configured above. If you are using CDCK hosting this is automatically handled for you. If you are self-hosting check the self-hosting guide.

  • ai_embeddings_models: Every model enabled here be used to generate embeddings from user posts.

  • ai_embeddings_semantic_suggested_model: Model that will be used to semantic suggested topics. Model picked here must be enabled in the ai_embeddings_models setting too.

  • ai_embeddings_generate_for_pms: If PMs should also have embeddings generated automatically when created.

  • ai_embeddings_semantic_related_topics_enabled: Shows related topics at the bottom of the topic page. This will add an extra block between the topic last post and the suggested topics with topics who are semantic related to the current topic.

  • ai_embeddings_pg_connection_string: Database connection string for the PostgreSQL instance that will store embeddings. If you are using CDCK hosting this is automatically handled for you. If you are self-hosting check the self-hosting guide.

  • ai_openai_api_key: OpenAI API key. Necessary if you want to use OpenAI to generate embeddings. If so, don’t forget to pick text-embedding-ada-002 in ai_embeddings_models and OpenAI - gpt-3.5-turbo in ai_embeddings_semantic_search_hyde_model.

17 Likes

Great work, thanks first of all, but I can’t see similar topics under the topics, somehow, my settings are like this, I added an openai key. Semantic search works, but how can I show similar articles under topics?

If you want to use OpenAI for embeddings you must set ai embeddings model to text-embedding-ada-002.

1 Like

How are the jobs to generate embeddings scheduled? From the code it seems like embeddings are only generated when the page is viewed and embeddings are missing. Is there a way to generate embeddings for the whole site when turning the feature on?

2 Likes

You can also run rake ai:embeddings:backfill to generate embeddings for all topics eagerly.

6 Likes

Suggestion

Sometimes reading a topic one knows most of the noted background but there are also some mentions that are not known. While there is summarization for summarizing an entire topic up to that point what would also be of help would be an AI option that inserts a glossary for the topic as a post near the top and updates it if a user selects a word or phrase that it wants the AI to include in the glossary.


Today in reading this topic there was one reference I did not recognize so looked it up and added a reply with a reference for it. While I know the remaining references I am sure there are others, especially those new to LLMs and such, that would have no idea of many of the noted references and if the AI could help them they would visit the site much more often.

While I know what RAG means in this starting post, how many really know that?

What is RAG (Click triangle to expand)

How do domain-specific chatbots work? An Overview of Retrieval Augmented Generation (RAG)


Note: Did not know with which topic to post this but since it needed embeddings to work posted it here. Please move this if it makes more sense elsewhere or as the Discourse AI plugin changes.

Are embeddings the only variable when determining “Related Topics”? Or are there any other factors that are considered (e.g. author, topic score, topic age, category, etc)?

3 Likes

Only the embeddings, but those contain the title, category, tags and posts. There is a site setting to remove closed topics from the results too.

5 Likes

7 posts were split to a new topic: Is full page semantic search only in English?

2 posts were split to a new topic: Differences in search latency between AI semantic and keyword search

I wish I found this a few months ago. I already created embeddings using bge-small-en-v1.5 and hosted them in an external database.

I will see if it can be shoehorned into this ‘standard’ set-up!

I find a little bug in the recent version leading to rake ai:embeddings:backfill failed:

root@nbg-webxj:/var/www/discourse# rake ai:embeddings:backfill
rake aborted!
NameError: uninitialized constant Parallel (NameError)

  Parallel.each(topics.all, in_processes: args[:concurrency].to_i, progress: "Topics") do |t|
  ^^^^^^^^
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:27:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => ai:embeddings:backfill
(See full trace by running task with --trace)

I suspect the culprit is that the parallel gem is neither installed in this plugin, nor in Discourse core(only find one in the if ENV["IMPORT"] == "1" block: gem "parallel", require: false).

I find the ruby-progressbar gem also required to perform rake ai:embeddings:backfill.

I make a simple PR on Github:

2 Likes

Note to others that this rake method seems to have been demoted/semi-deprecated since per Falco on GitHub:

Thanks for the PR @fokx, but I’ve left those out unintentionally as the rake task fell out out favor and should only be used in rare occasions by experienced operators who can easily install those out of band.

Is the semantic search option no longer shown in that dropdown and instead comprehended or enabled through the AI toggle?

1 Like

Can you confirm for me if the embeddings will only work on posts after installing or will it also allow us to semantic-search all historical posts? I’m hoping the latter! Thanks.

1 Like

It’s the later, as it will automatically backfill embeddings for all your content.

2 Likes