How to enforce exact matches when searching?

Support

AI-generated summary

Exact Match Search Issues in Discourse

Nacho_Caballero is experiencing issues with exact match searches in Discourse. Despite using quotes, search results include matches with similar but not identical words. Initially suspecting accented characters or stop words, Nacho_Caballero has tried modifying the Spanish stopwords dictionary.

To reproduce the issue, tgxworld requested a sample text and search term, which Nacho_Caballero provided. However, the issue only occurs with a large number of posts (around 68K).

Yashi12 inquired about the search algorithm used by Discourse, and Nacho_Caballero assumed it’s the default full-text search Postgres algorithm. Using the ‘simple’ dictionary might provide exact matches without stemming or ignoring stop words.

Overgrow also reported similar issues with exact match searches, and codinghorror suggested using quotation marks to search for exact phrases, but this does not always yield the desired results.

1

I’ve noticed that for some searches, I get no results:

CleanShot 2021-06-28 at 07.00.30@2x
CleanShot 2021-06-28 at 07.00.30@2x2392×346 12.7 KB

But if I narrow the search, for example to a specific topic, I do get the result I’m looking for:

CleanShot 2021-06-28 at 07.01.06@2x
CleanShot 2021-06-28 at 07.01.06@2x2382×916 161 KB

I initially thought it might have something to do with the accented character, but I’ve done several tests and the same thing happens with unaccented characters.

My second thought was that it might be due to stop words, but I’ve changed app.yml to truncate the Spanish stopwords dictionary:

ls -l /usr/share/postgresql/13/tsearch_data/spanish.stop
-rw-r--r-- 1 root root 0 May 12 06:37 /usr/share/postgresql/13/tsearch_data/spanish.stop
                       ☝️

Are there any settings I could tweak to make sure exact matches are always found?

3

Hi @Nacho_Caballero

Can you provide me with a sample text and search term that can reproduce the problem? I can see it in the image but only the excerpt can be seen.

4

Thanks, Alan.

Here are the contents of that specific post:

En efecto, no se usa después del “pienso que” porque lo que estás afirmando que piensas, se trata de una realidad, y no de una posibilidad.

Por otro lado, utilizamos subjuntivo cuando utilizamos esta misma estructura, pero en negativo: “No pienso que haya mucha gente en el mercado hoy”.

Espero que esto responda a tu respuesta, pero si necesitas más detalles, por favor dime, y con gusto te ayudaré.

¡Saludos! :smiley:

However, to reproduce the behavior here, you’d have to set the locale to Spanish and truncate the dictionary :sweat_smile:

I’ve tested it on a development Discourse site I have running locally and I’m not able to reproduce the behavior, but I think it’s because the site is practically empty. I don’t think this issue comes up until you have a lot of posts (I currently have around 68K).

I could try other things related to the postgres dictionary, but I’m not sure how to enter my docker image when running a development installation. Is there an equivalent to launcher enter app?

Ideally, there would be a way to ignore stop words. For example, to look for this sentence in your message:

CleanShot 2021-06-29 at 09.12.42@2x
CleanShot 2021-06-29 at 09.12.42@2x944×730 49.8 KB

5

@Nacho_Caballero ,Hey can u please let me know currently which search algo is Discourse using, it will help me in looking into the issue.

6

Thanks, Yashi.

I assume Discourse uses the default full-text search Postgres algorithm. I think the only way to provide exact matches when using quotes (without stemming and without ignoring stop words) would be using the ‘simple’ dictionary :thinking:

8

Thankyou, @Nacho_Caballero.

I will look over Postgres full text algo.

10

Is it possible to search for exact match?

Example:

At the moment, search for “flow” unfortunately returns “flower” as the highest relevance ..

11

Try searching with quotation marks, e.g. search for “flow” not flow.