Zelf-hosten van Embeddings voor DiscourseAI

Documentatie Zelf hosten
, ,
1

The Discourse AI plugin has many features that require embeddings to work, such as Related Topics, AI Search, AI Helper Category and Tag suggestion, etc. While you can use a third-party API, like Configure API Keys for OpenAI, Configure API Keys for Cloudflare Workers AI or Configure API Keys for Google Gemini, we built Discourse AI from the first day to not be locked into those.

Running with HuggingFace TEI

HuggingFace provides an awesome container image that can get you running quickly.

For example:

mkdir -p /opt/tei-cache
docker run --rm --gpus all --shm-size 1g -p 8081:80 \
  -v /opt/tei-cache:/data \
  ghcr.io/huggingface/text-embeddings-inference:latest \
  --model-id BAAI/bge-large-en-v1.5

This should get you up and running with a local instance of BAAI/bge-large-en-v1.5, a very good performing open-source model.

You can check if itā€™s working with

curl -X POST \
  'http://localhost:8081/embed' \
  -H 'Content-Type: application/json' \
  -d '{ "inputs": "Testing string for embeddings" }'

Which should return an array of floats under normal operation.

Making it available for your Discourse instance

Most of the time, you will be running this on a dedicated server because of the GPU speed-up. When doing so, I recommend running a reverse proxy, doing TLS termination, and securing the endpoint so it can only be connected by your Discourse instance.

Configuring DiscourseAI

Discourse AI now uses a fully configurable embedding definition system, similar to how LLMs are configured. To set up your self-hosted endpoint:

  1. Navigate to Admin ā†’ Plugins ā†’ Discourse AI ā†’ Embeddings.
  2. Click New to create a new embedding definition.
  3. Select a preset that matches your model (e.g. bge-large-en, bge-m3, or multilingual-e5-large), or choose Configure manually for any other model.
  4. Set the URL to point to your self-hosted TEI server (e.g. https://your-tei-server:8081).
  5. Use the Test button to verify connectivity before saving.
  6. After saving, set ai_embeddings_selected_model to your new embedding definition.

Once configured, Discourse will automatically backfill embeddings for existing topics via a scheduled background job. If you have a large backlog, you can increase the hidden setting ai_embeddings_backfill_batch_size (default: 250) to process topics faster.

Last edited by @Falco 2025-03-21T17:34:12Z

Check documentPerform check on document:
11 likes
Discourse AI - Self-Hosted Guide
Settings for Hugging Face bge-large-en embeddings? -> RAG bots unresponsive!
Weekly Summary of AI topics
Weekly Summary of AI topics
2

The model bge-m3 should work for multilingual (or not english) sites?

3

Yes, I played with it the week it got silently shared on GitHub and it works well. Still waiting to see how it lands on the MTEB leaderboars, as it wasnā€™t there last I looked.

That said we have large hosted Discourse instances using the multilingual the plugin ships, e5, and it performs very well.

1 like
4

Thanks, did you have plans to enable open-source custom endpoints for embeds? Iā€™m trying to use this models on Huggingface.

5

Sorry I donā€™t understand what you are trying to convey here. This topic is a guide on how to run open-source models for Discourse AI embeddings.

6

Oh, sorry about that. Iā€™m trying to use an open-source model from HuggingFace custom endpooint and I wonder if thatā€™s possible or itā€™s on the plans to enable at near future :slight_smile:

7

To check if itā€™s working, the following command works for me (with BAAI/bge-m3 model):

curl -X 'POST' \
  'http://localhost:8081/embed'\
  -H 'Content-Type: application/json' \
  -d '{ "inputs": "Testing string for embeddings"}'

BTW, you can also use the Swagger web interface at http://localhost:8081/docs/.

2 likes
8

This is also a nice embeddings server:

1 like
9

To save space, is it possible to use quantized embeddings? Iā€™d like to use binary quantized embeddings to really cut down the storage size. Having done some tests, I get >90% performance with 32x less storage!

1 like
10

We are storing embeddings using half precision (half storage space) and using binary quantization for indexes (32x smaller) by default as of a few weeks ago, so just updating your site to latest should give you ample disk usage reduction.

3 likes
11

Could you please also add:

to the supported embedding models?

12

We plan on making embeddings configurable the same way we did with LLMs, so any model will be compatible soon.

4 likes
13

If anyone else has problems with endpoints on the local network e.g. 192.168.x.x - it seems these are blocked by discourse (presumably for security reasons) and the block needs to be bypassed. Lost some hours figuring that one out!

1 like
14

@Falco that would be great. In the interim, if I wanted to have a stab at adding in a new embedding model, do I just need to add:

 lib/embeddings/vector_representations/mxbai-embed-xsmall-v1.rb
 lib/tokenizer/mxbai-embed-xsmall-v1.rb
 tokenizers/mxbai-embed-xsmall-v1.json

and modify lib/embeddings/vector_representations/base.rb to include the new model, or is there something else I need to change too?

15

@Falco I tried my hand at adding the model and sent a pull request. Apologies if I did something wrong as Iā€™m not really a SW developer. I hoped you could maybe look over it and see if it is OK for inclusion.

Unfortunately, I was not able to get it working with TEI. I could get the all-mpnet working with TEI, but I think thereā€™s something wrong with what I have done to get mxbai working.

BTW, any chance of supporting https://github.com/michaelfeil/infinity as an embedding server?

EDIT: I see this is going to be messy as the HNSW indexes in the database seem to be hardcoded so new models need to be appended at the end to avoid disrupting the ordering and each new model needs to add its own index.

16

I really recommend waiting a couple of weeks until we ship support for configurable embeddings.

This should work fine when we ship configurable embeddings, but out of curiosity what would that bring over GitHub - huggingface/text-embeddings-inference: A blazing fast inference solution for text embeddings models ?

17

I havenā€™t kept up with TEI so wonā€™t mention the advantages that I havenā€™t tested recently, but of then things I saw recently:

  • Hardware support: infinity has better GPU support than TEI
  • infinity server can host multiple embedding models in a single server (unless I missed this in TEI)

Itā€™s very nice. If you havenā€™t tried it, you should take a look!

1 like
18

A friend just DMā€™ed me this thread.

Some Pro/Conā€™s:

  • infinity supports multi-modal embeddings (aka send images/audio) to the
  • amd gpu support
  • multiple models supported in the same container (control the model via model param).
  • more dtypes e.g. int8 quantization of the weights (mostly this is irrelevant, activation memory is larger)
  • new models often come out via ā€œcustom modeling codeā€ shipped in the huggingface repo. Infinity reads this pytorch code if needed. This will help you avoid ā€œcan you support xyz modelsā€ on a ongoing basis)
  • more models supported (e.g. debertav2 for mixedbread)

Cons:

  • cold start time of TEI is better
3 likes
19

Hi Michael :wave:

@roman has been busy restructuring our embedding config at:

We should be done very very soon, once that is done adding support for inifinity should be trivial.

I still think a lot about multi model embedding, it gives you a shortcut when trying to do RAG on PDFs cause you just process it into images and embed each image avoiding need for OCR or expensive Image to text powered by LLM.

Once we get this PR done we will be more than happy to add infinity support (and multi model support) into the embedding config.

Thanks for popping in :hugs:

4 likes
20

I wonder whether building litellm support might offer a shortcut as then you benefit from all the models supported via litellm. Other projects see to embed this.