We have considered various integrations including sphinx, melli, elastic, solr/lucene but they come at a cost. Hosting another process to run indexing, risking out-of-date indexes, complexity… etc all are not free.
I would like to see how much mileage we get out of PG prior to exploring any other options and keep them as a last resort.
Very interesting problem, yes, they are (and always have been) de-prioritized. I think at a minimum we can look at adding a site setting to discourse-solved to allow admins to decide what to do in these cases (prioritize/deprioritize/neutral etc.)
Unfortunately, postgres is not adapted as a search engine. And meilisearch has fantastically low memory consumption and limitless search possibilities. The overhead for the server compared to ruby will simply be invisible.
This is not a trivial problem. Our search contains enormous amounts of dimensions and has lots of params, it joins directly into postgres tables.
With an external search provider we need to worry about “synchronization”.
A topic is closed on Discourse → notify engine
A post is deleted → notify engine
A like is made → notify engine
A topic is split or merged → notify engine
The list goes on, including building multiple indexes (users/posts/topics/categories)
That said, given the right investment this is not necessarily insurmountable, but it is an enormous task and there is no proof of concept out there showing how much better it would be. It is nice that melli has a typo ranker, and many other features no argument there. But integrating it is not free at all.
As a rough estimate I would think there is about 3 months work building tight and robust integration into mellisearch. Maybe even 6 months if we were to design Discourse in such a way that search engine is “pluggable”
Note that we do support algolia integration here: https://discourse.algolia.com/ it is not quite rock solid, and you can see that the entire advanced search is omitted from the implementation.
After sometimes now I asked what my the most active users thought (thinked ) about searching — I never told it got some steroids.
Everyone said exactly same thing; they haven’t think it but because I asked they realize they found now relevant hits much easier, in most of cases right away,
One part of Discourse is acting as commenting system of WordPress. No, I don’t get more comments (nothing is so overrated than commenting of blogs) but it has showed existence (is it spelled that way?) of the forum. Nowadays I have a handful users who are using Discourse as a search engine, They don’t comment but they search what they are looking for from WordPress via Discourse topics and go back to the blog. Sure, tag-system helps a lot too. And WordPress is missing both: effective searching and working tagging,
I don’t know if I should post this in praise instead, but I wanted just tell that I’m quite pleased how this new and improvent search works.
Not sure if this was a problem before but I noticed many system-created posts showing up in search results. Maybe an edge case more noticeable here on meta, but I wouldn’t expect system-messages to be relevant to search.
Example result when searching for terms like “automatically closed”: