Refinements to search being tested on meta

[sam (Sam Saffron)]

Yes, but you need to run rake search:reindex as well

[volanar]

Have you thought about improving the search using meilisearch? This requires very few resources and can be included in the docker build.

[Falco]

@Tris20

The one thing I would love to see a bit of improvement on is common spelling mistakes. Some search engines have spoiled me to the point that I will lazily slap the keys until a mixture of letters that only vaguely resemble to original word are in the search bar . I don’t expect that to be matched, but improvements in that area would be a great step forward.

We have started experiments in this area by

@sam

We plan to experiment with “fuzzy” search for mention autocomplete. (allow you to skip a letter for example)

First experiments are limited to user / group search, but if all goes well it can expanded further.

[sam (Sam Saffron)]

@volanar

Have you thought about improving the search using meilisearch?

We have considered various integrations including sphinx, melli, elastic, solr/lucene but they come at a cost. Hosting another process to run indexing, risking out-of-date indexes, complexity… etc all are not free.

I would like to see how much mileage we get out of PG prior to exploring any other options and keep them as a last resort.

@loginerror

Quick question about the above. What about auto-closed topics with a solution?

Very interesting problem, yes, they are (and always have been) de-prioritized. I think at a minimum we can look at adding a site setting to discourse-solved to allow admins to decide what to do in these cases (prioritize/deprioritize/neutral etc.)

[volanar]

@sam

I would like to see how much mileage we get out of PG prior to exploring any other options and keep them as a last resort.

Unfortunately, postgres is not adapted as a search engine. And meilisearch has fantastically low memory consumption and limitless search possibilities. The overhead for the server compared to ruby will simply be invisible.

[sam (Sam Saffron)]

@volanar

The overhead for the server compared to ruby will simply be invisible.

This is not a trivial problem. Our search contains enormous amounts of dimensions and has lots of params, it joins directly into postgres tables.

With an external search provider we need to worry about “synchronization”.

• A topic is closed on Discourse → notify engine
• A post is deleted → notify engine
• A like is made → notify engine
• A topic is split or merged → notify engine

The list goes on, including building multiple indexes (users/posts/topics/categories)

That said, given the right investment this is not necessarily insurmountable, but it is an enormous task and there is no proof of concept out there showing how much better it would be. It is nice that melli has a typo ranker, and many other features no argument there. But integrating it is not free at all.

As a rough estimate I would think there is about 3 months work building tight and robust integration into mellisearch. Maybe even 6 months if we were to design Discourse in such a way that search engine is “pluggable”

Note that we do support algolia integration here: https://discourse.algolia.com/ it is not quite rock solid, and you can see that the entire advanced search is omitted from the implementation.

[volanar]

@sam

As a rough estimate I would think there is about 3 months work building tight and robust integration into mellisearch. Maybe even 6 months if we were to design Discourse in such a way that search engine is “pluggable”

I’m willing to bet that with such a large community of discourses like discourse, it can be much faster, no more than three months

[Jagster (Jakke Lehtonen)]

After sometimes now I asked what my the most active users thought (thinked ) about searching — I never told it got some steroids.

Everyone said exactly same thing; they haven’t think it but because I asked they realize they found now relevant hits much easier, in most of cases right away,

One part of Discourse is acting as commenting system of WordPress. No, I don’t get more comments (nothing is so overrated than commenting of blogs) but it has showed existence (is it spelled that way?) of the forum. Nowadays I have a handful users who are using Discourse as a search engine, They don’t comment but they search what they are looking for from WordPress via Discourse topics and go back to the blog. Sure, tag-system helps a lot too. And WordPress is missing both: effective searching and working tagging,

I don’t know if I should post this in praise instead, but I wanted just tell that I’m quite pleased how this new and improvent search works.

[sam (Sam Saffron)]

@Jagster

Everyone said exactly same thing; they haven’t think it but because I asked they realize they found now relevant hits much easier, in most of cases right away,

Wow thanks, this certainly makes me feel really good! We have a PR in the oven now and we should be rolling out the changes globally really soon.

[mattdm (Matthew Miller)]

@sam

We have a PR in the oven now and we should be rolling out the changes globally really soon.

Sorry if I’m being obtuse — should this be active on hosted sites (with latest deploy)? The release announcement points here, but this talks about a hidden setting — is that hidden setting on?

[mcwumbly (Dave McClure)]

You do not need to do anything:

@sam

@mcdanlj

it looks like prioritize_exact_search_title_match is still hidden. Is this something we should expect to use? Are we supposed to run rake search:reindex after deploy as I see in this comment in the thread?

No and no

We amended the default prioritize_exact_search_title_match is now default true.

We also bumped up our index version to version 5.

This means that every 2 hours 10,000 topics and 20,000 of the newest posts with an old index version will be reindexed.

You can rush the process by running rake search:reindex but there is no need really, it will happen automatically in the background anyway. (it probably did already for most)

I’ll update the original post with a note.

[Luke_Cousins]

Thanks for the fantastic update. For us being able to define search synonyms would be a huge improvement Thank you.

[dsims]

Not sure if this was a problem before but I noticed many system-created posts showing up in search results. Maybe an edge case more noticeable here on meta, but I wouldn’t expect system-messages to be relevant to search.

Example result when searching for terms like “automatically closed”:

[mattdm (Matthew Miller)]

I can’t reproduce that here.

[Moin]

I can reproduce that; if you sort them by latest post instead of relevance, there are a lot of system messages in the results.

[mattdm (Matthew Miller)]

Ah, yeah I see that then. It isn’t everything, but more than reasonable. It does seem like these messages should be excluded from search