Discourse AI - Embeddings

[Falco]

@Overgrow

Do I understand it correctly that backfill is when the vectorization happens?

Yes.

@Overgrow

When switching between models, do the vectors need to be recalculated

Yes, each model produces different vector representations.

@Overgrow

It’d be useful to know how the costs of using the OpenAI API stack up against investing in a GPU-powered server with opensource solution

It’s basically one call per topic, so very easy to calculate.

@Overgrow

Is there a formula or any way to estimate the number of tokens used?

If most of your topics are long, they will be truncated to 8k tokens, otherwise they will use your topic length.

@Overgrow

We’re only using the API to vectorize posts, not for calculating vector distances, right?

Yes.

@Overgrow

I assume that for both related topics and AI-powered search, all posts need to be vectorized only once, so I can calculate the total number of words in posts table and derive the number of tokens needed. The same process would apply to the daily addition of posts. I’m neglecting the search phrases for now.

Both work at the topic level, so one per topic.

[RGJ (Richard - Communiteq)]

It seems that this documentation topic is out of date since this commit, as well as this documentation topic.

[Falco]

Indeed. @Saif can you update here?

[Saif (Saif Murtaza)]

@Falco

Indeed. @Saif can you update here?

The OP has now been updated

[satonotdead]

@falco

I’ve left those out unintentionally as the rake task fell out out favor and should only be used in rare occasions by experienced operators who can easily install those out of band.

May I know how to properly add the gems without forking the plugin with the suggested PR?

I’m trying the scale-to-zero feature on HuggingFace and I just need to use the rake task for backfill embeddings.