Oh, I see now, thanks for the explanation, that’s what I was missing.
Btw, I know often there’s reranking in more advances RAG. Is there something like reranking in how Discourse handles is? Do you think adding a reranking step would have any positive effect?
reranking is a work-in-progress.
@Falco / @Roman implemented a basic reranker in discourse:
This is used in semantic search and RAG. However it is quite hidden and not easy to configure.
I think the medium term plan here (which we discussed with @awesomerobot ) was to move from LLM terminology to Models … and maybe do a bit of UI unification so you can define embedding/rerankers and llms in a single interface.
For now we only support a very specific hugging face reranker api.
It certainly improves quality of results.
Awesome!
So currently this is off by default, and can’t be configured easily on a hosted (pro) plan?
not sure, lets wait for @Falco to answer.
Hi @Falco
is there any way to enable the reranker for testing or maybe a timeline?
We just deployed a big improvement to the underlying tech that powers semantic search in Discourse in DEV: Re-introduce PG Vector 0.8.0 upgrade by romanrizzi · Pull Request #35233 · discourse/discourse · GitHub.
Can you retry your use case now, it’s already deployed to your site. My expectation is that this will make RAG better without the need for a reranker.
We still want to make the re-ranker widely available, but we are waiting for some upstream changes to land first.
Thanks Falco!
Will this work only for the semantic search over Discourse topics, or also for RAG documents uploaded to a persona? From my own experience, the AI works quite well with forum topics (i.e. on ask.discourse), but not that well with uploaded docs to a persona (or I haven’t found the best formula yet).
It affects all uses of embeddings in Discourse, including Related Topics, RAG, Search, Composer category and tag suggestions, etc.
Hey @tpetrov, did the results improve with the new embeddings model?
Hey Falco, I’m sorry, I haven’t had time to test extensively, so cannot really say.
I guess there’s no way to sitch between the two to compare?
Not now, the old and new models were available for a couple of months, but we recently retired the old ones in our hosting.