Having played with it a little now, some observations:
- The results are different to regular search. Not always better because sometimes a keyword/relevance search actually seems to get the best hit when you know the keywords to search for, but …
- It does return a wider set of results and these are sometimes really useful.
- It’s currently really slow for me. I realise this is somewhat unavoidable because there is an LLM generation step in front of the retrieval so not sure this can be fixed, but worth knowing. It seems slower than a short gpt-3.5 call would be…
Some UI points:
- It’s not always obvious in the returned results which part is relevant. In an app I’m working on I’m chunking documents quite small before creating embeddings (sentences/paragraphs) and this means when searching//retrieving it’s possible to colour each sentence according to the semantic similarity. This would be a bit like highlighting the keywords from the search, but would appear a bit like a heat map with semantically similar parts coloured hot and dissimilar cold.
- It’s annoying to have to click to expand the semantic results.
- Have you thought of ways to combine keyword and semantic results? Would it be possible to choose ‘similarity’ or ‘relevance’ as the sort order for result set? That way, if you picked ‘similarity’ then you could start by presenting keyword-based results and insert hyde-retrieved documents into the list as they arrive.
- It would be really interesting (to me at least) to be able to see the hypothetical document used to do the similarity matching. I can imagine sometimes wanting to edit this document… and because Cosine similarity is relatively cheap (compared to the LLM generation call) it would still be quite fast in the UI to update results as the user updates their query/hypothetical document.
All in all — this is really cool, thanks! Will be great when this is implemented such that the chatbot can use the results.
B