The search should match fancy characters with their "regular" equivalent

Canapin · August 25, 2020, 11:05pm

I’ve copy-pasted a topic title (as it is displayed, with fancy entities) in the search
New Lowe’s commercial with UniGeezer

No result:

I replace the fancy apostrophe by the “regular” one in the search field:
New Lowe's commercial with UniGeezer

Now the topic appears.

My suggestion is that the search should match every fancy character with the original one.

codinghorror · August 26, 2020, 2:19am

Good point, how should we handle this @sam?

TheBestPessimist · August 26, 2020, 4:29am

What about diacritics?

sam · August 26, 2020, 8:00am

We have some normalization for diacritics already so maybe we can also correct this in a similar path.

@tgxworld can have a think about it.

tgxworld · December 22, 2020, 3:03am

@Canapin Are you still able to reproduce this? I tried to reproduce this locally but couldn’t. The apostrophe is stripped from the search data so it should not have any effect on search.

discourse_development=# SELECT TO_TSVECTOR('english', 'New Lowe’s commercial with UniGeezer') @@ PLAINTO_TSQUERY('english', 'New Lowe’s commercial with UniGeezer');
 ?column? 
----------
 t
(1 row)

Are you able to point me to the site which you’re facing this problem so that I can get a repro? Thank you!

Canapin · December 23, 2020, 12:27am

I still have the issue, and it’s when I search for the exact string (wrapped by "):

https://unicyclist.com/search?q=%22New%20Lowe%E2%80%99s%20commercial%20with%20UniGeezer%22

vs

https://unicyclist.com/search?q=%22New%20Lowe%27s%20commercial%20with%20UniGeezer%22

tgxworld · December 23, 2020, 3:55am

Thank you for the repro. This basically affects search for exact terms when the search terms are wrapped in ". The problem here is that the real title of the topic is actually New Lowe's commercial with UniGeezer but the fancy title is New Lowe’s commercial with UniGeezer. When we do a search for exact terms, we’re only matching the given terms to the topic’s title and not the fancy title.

The difficulty here is that we can’t just replace ’ with ' unconditinally becasue a topic title with ’ in it will end up not matching. I kind of unsure what we can do here because we’re displaying different characters on the client side when displaying the topic title.

@gerhard @sam It seems like you have tackled this issue around quoting before, any ideas what we can do here? To be honest though, it is an edge case that will affect a very small portion of search queries. I’m inclined to just pun on this.

sam · December 23, 2020, 4:26am

This is no laughing matter!

I guess we could normalize ’ to ' in the index and search term. But I am honestly not sure it is worth a giant effort fixing this.

tgxworld · December 23, 2020, 5:42am

This is not related to the search index. For exact matches, we match it against Post#raw and Topic#title:

github.com

discourse/discourse/blob/755627caa512058ebee332f40f9743024bb262f3/lib/search.rb#L942


    # A is for title
    # B is for category
    # C is for tags
    # D is for cooked
    weights = @in_title ? 'A' : (SiteSetting.tagging_enabled ? 'ABCD' : 'ABD')
    posts = posts.where(post_number: 1) if @in_title
    posts = posts.where("post_search_data.search_data @@ #{ts_query(weight_filter: weights)}")
    exact_terms = @term.scan(Regexp.new(PHRASE_MATCH_REGEXP_PATTERN)).flatten
    exact_terms.each do |exact|
      posts = posts.where("posts.raw ilike :exact OR topics.title ilike :exact", exact: "%#{exact}%")
    end
  end
end
@filters.each do |block, match|
  if block.arity == 1
    posts = instance_exec(posts, &block) || posts
  else
    posts = instance_exec(posts, match, &block) || posts
  end

sam · December 23, 2020, 5:45am

I see, yeah … no easy solution here at all, I think this is just a nit we have to live with.