Search a term in Japanese

Thank you for your reply.

  1. A sample paragraph here in katakana
    通報テスト9,通報テスト11,通報テスト8…etc

  2. A sample search term that you have that is not working
    テスト
    The “テスト” is not working.


    But the “通報” or “通報テスト” seems to be working correctly.

  3. Confirmation that your site locale is in Japanese or that search tokenize chinese japanese korean is enabled
    Yes, I have confirmed that both settings are set correctly.


1 Like

An incredible thing happened. After changing the ‘min search term length’ from the default value of 2 to 1, we are now able to search for katakana. I don’t know why, but is this setting relevant?

1 Like

I can repro this and it is mainly due to a combination of

https://github.com/discourse/discourse/blob/e8a842ab8cbbabe92fe33cfc4bbe5f839d4543e9/lib/search.rb#L66-L69

and

https://github.com/discourse/discourse/blob/e8a842ab8cbbabe92fe33cfc4bbe5f839d4543e9/lib/search.rb#L242-L243

The term テスト is converted to テ ス ト after going through CppjiebaRb and this trips the min_search_length protector we have.

@sam This is tricky to fix because we need a proper tokenizer for Japanese to resolve search issues like this for good. We can do tweaks here and there but it is going to be a game of wack a mole.

3 Likes

I don’t think there exists a proper Japanese segmentar we can use.

I think the best thing to do here is simply tone down these defaults to 1.

https://github.com/discourse/discourse/blob/580383dff342a9a12f2270a8224b91c12f0e6ca7/config/site_settings.yml#L1837-L1844

Otherwise we are banning people from searching for house in Japanese which seems reasonable (家) … we allow people to search for house in English.

2 Likes