Search Improvements in 2.3

For the upcoming 2.3 release, we’ve added a new feature and made several fixes to the way we index posts for search that will make search results better.

1. Search Priorities for Categories


Each individual category can be configured with a search priority which you can find under the Settings tab when creating/editing a category. Five levels for prioritization have been added: ignore, very low, low, high and very high. The levels work by multiplying a pre-configured weight to the search ranking for each result and can be configured via hidden site settings in your console.

As an example, configuring a category’s search priority to very_high will boost its search ranking by 40% while a search priority of very_low will reduce its search ranking by 40%. Setting a category’s search priority to ignore will remove it from the search results. However, you can still search for posts within the ignored category by scoping the search to the category via advanced search. Do take note that search priority is not inherited which means that a sub-category will still be searchable even if the parent category has been configured to be ignored.

2. Improvements and Fixes to Search Results

  1. Search relevance has been updated to consider the document length in the ranking. Previously, search ranking was ranked by the most number of matches for the given search term. This was problematic for search results as we noticed that longer posts were more likely to be ranked higher because of the tendency to produce a higher number of matches. As a result, we’ve made the switch to consider the document length when ranking search results.
    FIX: Relevance search will now consider document length in ranking. · discourse/discourse@e87ca59 · GitHub

  2. Improve the quality of the raw data that we use to generate the search index. PERF: Improve quality of `PostSearchData#raw_data`. (#7275) · discourse/discourse@cfd5078 · GitHub

    • URLs in posts were sometimes incorrectly tokenized and indexed twice leading to terms within links having a higher ranking.
    • Content for lightboxes in posts were polluting search results with the image’s metadata.
  3. Empty posts such as moderator actions or small post actions (closing, assigning) are no longer part of the search index. This change leads to a smaller index and less noise in search results. FIX: Don't index posts with empty `Post#raw` for search. (#7263) · discourse/discourse@daeda80 · GitHub

  4. Smaller search index by deleting posts of trashed topics from the index. PERF: Delete search data of posts from trashed topics periodically. (… · discourse/discourse@d151425 · GitHub

  5. When a post is moved to a different topic, the search data for the post was not updated and would lead to the post incorrectly appearing in searches that matched the category of the old topic. FIX: Reindex post for search when post is moved to a different topic. · discourse/discourse@d808f36 · GitHub

  6. Smaller payload for search results by excluding the cooked version of the posts. PERF: Reduce number of queries and size of payload when searching. · discourse/discourse@03c6b22 · GitHub

  7. The blurb for posts in search results was broken when searching for an exact phrase and lead to missing search term highlights on the client side. FIX: Post blurb incorrect when search contains a phrase match. · discourse/discourse@dae0bb4 · GitHub

Do let us know if you’re search results have improved after these changes. We would also like to know if you think your search results have become worse so that we can continue to refine it. Thank you all!

42 Likes

Is the age of a post part of the weight in ranking search results? Information gets stale fairly quickly in our forum, so it would be nice to have a way to reduce the relevance of older postings without actually eliminating them.

11 Likes

Not yet, but it is an interesting idea, even in the weaker form of simply factoring in the date the topic was last touched

7 Likes

I would love to see a way to “pin” topics in search results so they appear first. That way common searches things can be lead directly to our tutorials.

You do this by changing search priority for the category those topics are in.

My “issue” is that those topics are spread through all forum categories on contextual basis. Hence I can’t use the category-based approach.

Then you are stuck because there is no random per topic way to do this, and there never will be.

Maybe search priorities for tags could be a thing eventually? Although not as straight forward since topics can have multiple tags…

Being able to weight categories is a great feature anyway! I already have several spammy special-use categories on my forum that will be nice to push down a bit in search.

6 Likes

Yes, I see this happening at some point.

9 Likes

That would be awesome! That way blog posts can be tagged as such and prioritized for search…

1 Like

I would love to see a way for certain tags to affect the search priority also.

4 Likes

Is there any plan to add this to 2.4.X? or 2.5?