Refinements to search being tested on meta

Have you thought about improving the search using meilisearch? This requires very few resources and can be included in the docker build.

3 Likes

7 posts were split to a new topic: Prioritizing closed or solved topics in search

We have started experiments in this area by

First experiments are limited to user / group search, but if all goes well it can expanded further.

8 Likes

We have considered various integrations including sphinx, melli, elastic, solr/lucene but they come at a cost. Hosting another process to run indexing, risking out-of-date indexes, complexity… etc all are not free.

I would like to see how much mileage we get out of PG prior to exploring any other options and keep them as a last resort.

Very interesting problem, yes, they are (and always have been) de-prioritized. I think at a minimum we can look at adding a site setting to discourse-solved to allow admins to decide what to do in these cases (prioritize/deprioritize/neutral etc.)

16 Likes

Unfortunately, postgres is not adapted as a search engine. And meilisearch has fantastically low memory consumption and limitless search possibilities. The overhead for the server compared to ruby will simply be invisible.

3 Likes

This is not a trivial problem. Our search contains enormous amounts of dimensions and has lots of params, it joins directly into postgres tables.

With an external search provider we need to worry about “synchronization”.

  • A topic is closed on Discourse → notify engine
  • A post is deleted → notify engine
  • A like is made → notify engine
  • A topic is split or merged → notify engine

The list goes on, including building multiple indexes (users/posts/topics/categories)

That said, given the right investment this is not necessarily insurmountable, but it is an enormous task and there is no proof of concept out there showing how much better it would be. It is nice that melli has a typo ranker, and many other features no argument there. But integrating it is not free at all.

As a rough estimate I would think there is about 3 months work building tight and robust integration into mellisearch. Maybe even 6 months if we were to design Discourse in such a way that search engine is “pluggable”

Note that we do support algolia integration here: https://discourse.algolia.com/ it is not quite rock solid, and you can see that the entire advanced search is omitted from the implementation.

8 Likes

I’m willing to bet that with such a large community of discourses like discourse, it can be much faster, no more than three months

2 Likes

After sometimes now I asked what my the most active users thought (thinked :man_facepalming: ) about searching — I never told it got some steroids.

Everyone said exactly same thing; they haven’t think it but because I asked they realize they found now relevant hits much easier, in most of cases right away,

One part of Discourse is acting as commenting system of WordPress. No, I don’t get more comments (nothing is so overrated than commenting of blogs) but it has showed existence (is it spelled that way?) of the forum. Nowadays I have a handful users who are using Discourse as a search engine, They don’t comment but they search what they are looking for from WordPress via Discourse topics and go back to the blog. Sure, tag-system helps a lot too. And WordPress is missing both: effective searching and working tagging,

I don’t know if I should post this in praise instead, but I wanted just tell that I’m quite pleased how this new and improvent search works.

11 Likes

Wow thanks, this certainly makes me feel really good! We have a PR in the oven now and we should be rolling out the changes globally really soon.

11 Likes

Sorry if I’m being obtuse — should this be active on hosted sites (with latest deploy)? The release announcement points here, but this talks about a hidden setting — is that hidden setting on?

6 Likes

You do not need to do anything:

I’ll update the original post with a note.

9 Likes

Thanks for the fantastic update. For us being able to define search synonyms would be a huge improvement :pray: Thank you.

6 Likes

9 posts were split to a new topic: Can I exclude usernames from search

Not sure if this was a problem before but I noticed many system-created posts showing up in search results. Maybe an edge case more noticeable here on meta, but I wouldn’t expect system-messages to be relevant to search.

Example result when searching for terms like “automatically closed”:

4 Likes

I can’t reproduce that here.

3 Likes

I can reproduce that; if you sort them by latest post instead of relevance, there are a lot of system messages in the results.

4 Likes

Ah, yeah I see that then. It isn’t everything, but more than reasonable. It does seem like these messages should be excluded from search

3 Likes

2 posts were split to a new topic: Make howto a synonym for the #how-to tag

7 posts were split to a new topic: Making #documentation more prominent in Search

I think this is the best option.