Support for Vanilla Embedding Search

Feature
,
1

Would be nice to have just the simple semantic search available without having to go through the hyde model.

I have a document on how to set up a perforce client which we use for version control. When searching for version control, normal search doesn’t pop up because the page doesn’t mention the keyword.

When enabling the full page search I get a 500 error which always pops up when the AI result has 0 answers

image
image1163×532 16.7 KB

However when using the
discourse-ai/embeddings/semantic-search.json?hyde=false&q=Version%20Control end point, I do see the topic about perforce in the list.

Not sure if HyDE decides to filter it out or it’s because of the 500 error but would be nice if we can just have search using semantic + keyword as default.

1 Like
3

We had a search mode like this in the header quick search, with just asymmetric embeddings plus a re-ranking step, but it produced bad results in our testing using 2023 embedding models.

This week, we just updated our embedding model here on Meta to the current state of the art, and it’s very promising, so this is something we are definitely revisiting soon.

2 Likes
4

Sounds exciting! Can’t wait to try it out if it goes wide

1 Like