A discussion on search functionality in Discourse began with Canapin pointing out that searching for a topic with a fancy character (e.g., Lowe’s) doesn’t yield results, but searching with the regular equivalent (e.g., Lowe’s) does. Canapin suggested that the search should match fancy characters with their regular equivalents.
codinghorror agreed and asked for input from sam, who mentioned that some normalization for diacritics already exists. tgxworld investigated the issue but couldn’t reproduce it, later discovering that the problem occurs when searching for exact terms wrapped in double quotes.
tgxworld explained that the issue is due to the real title of the topic being different from the fancy title displayed on the client side. sam suggested normalizing ’ to ' in the index and search term, but tgxworld clarified that the issue is not related to the search index.
Ultimately, sam concluded that there is no easy solution, and this might be a “nit” that needs to be lived with.
I’ve copy-pasted a topic title (as it is displayed, with fancy entities) in the search
New Lowe’s commercial with UniGeezer
No result:
I replace the fancy apostrophe by the “regular” one in the search field:
New Lowe's commercial with UniGeezer
Now the topic appears.
My suggestion is that the search should match every fancy character with the original one.
Good point, how should we handle this @sam?
What about diacritics?
We have some normalization for diacritics already so maybe we can also correct this in a similar path.
@tgxworld can have a think about it.
@Canapin Are you still able to reproduce this? I tried to reproduce this locally but couldn’t. The apostrophe is stripped from the search data so it should not have any effect on search.
discourse_development=# SELECT TO_TSVECTOR('english', 'New Lowe’s commercial with UniGeezer') @@ PLAINTO_TSQUERY('english', 'New Lowe’s commercial with UniGeezer');
?column?
----------
t
(1 row)
Are you able to point me to the site which you’re facing this problem so that I can get a repro? Thank you!
I still have the issue, and it’s when I search for the exact string (wrapped by "):
https://unicyclist.com/search?q=%22New%20Lowe%E2%80%99s%20commercial%20with%20UniGeezer%22
vs
https://unicyclist.com/search?q=%22New%20Lowe%27s%20commercial%20with%20UniGeezer%22
Thank you for the repro. This basically affects search for exact terms when the search terms are wrapped in ". The problem here is that the real title of the topic is actually New Lowe's commercial with UniGeezer but the fancy title is New Lowe’s commercial with UniGeezer. When we do a search for exact terms, we’re only matching the given terms to the topic’s title and not the fancy title.
The difficulty here is that we can’t just replace ’ with ' unconditinally becasue a topic title with ’ in it will end up not matching. I kind of unsure what we can do here because we’re displaying different characters on the client side when displaying the topic title.
@gerhard @sam It seems like you have tackled this issue around quoting before, any ideas what we can do here? To be honest though, it is an edge case that will affect a very small portion of search queries. I’m inclined to just pun on this.
This is no laughing matter! 
I guess we could normalize ’ to ' in the index and search term. But I am honestly not sure it is worth a giant effort fixing this.
This is not related to the search index. For exact matches, we match it against Post#raw and Topic#title:
I see, yeah … no easy solution here at all, I think this is just a nit we have to live with.