Search a term in Japanese

סיכום מבוסס בינה מלאכותית

Users are reporting issues with Japanese text search in Discourse, particularly with Katakana (e.g., ‘トークン’, ‘テスト’) and some Kanji (e.g., ‘北側’, ‘真上’, ‘一般’) not returning results despite correct settings like search tokenize chinese japanese korean enabled and default_locale set to Japanese. kaito and SSS confirmed the problem persists even after rebuilding the search index using rake search:reindex Stephen. A workaround was found by reducing min search term length from 2 to 1, which enabled Katakana search for SSS. tgxworld traced the issue to CppjiebaRb tokenization splitting ‘テスト’ into ‘テ ス ト’, triggering the minimum length filter. sam proposed lowering the default min length to 1, acknowledging the lack of a robust Japanese tokenizer under permissive licenses. yashi suggested MeCab, but sam noted its GPL license is incompatible; instead, TinySegmenter (MIT licensed) is being considered, and yashi confirmed it correctly segments problematic terms, though its model quality may be limited. The core issue remains unresolved in v2.6, requiring either config tweaks or improved CJK tokenization.

It does not work.
Any solution to the problem?

Thank you.

Could you post a link to a post that should be found and the search term you are using?

Thank you for the reply.
I mean,
I am running own discourse on my server but I cant search in Japanese.

It’s hard to know why you can’t search without an example. If it’s a public forum, please post a link to a topic and tell us the search term that isn’t working. If it’s a private forum, can you copy & paste the raw post in here and also tell us the search term?

Also, which version of Discourse are you running? And is the default_locale set to Japanese?

It’s a private forum and all words I cannot search.
Like “トークン” , “トークン設計”.

The version is latest, 2.2.0.beta2.
And the default locale is Japanese.

Could you advice me if any updates on this topic?

This is our forum, the private one though.
https://token-lab.org/

Thank you.

There isn’t anything inherent blocking the search of japanese characters, for example from your post above:

Yes, seems to work here:

https://meta.discourse.org/search?q=%E3%83%88%E3%83%BC%E3%82%AF%E3%83%B3%E8%A8%AD%E8%A8%88

Thank you, hmm it doesn’t work in our forum. Let me check.

Is the search tokenize chinese japanese korean site setting enabled?

Toggling that and rebuilding the search index may help.

Does our first run setup wizard toggle this site setting when those languages are selected during initial setup? :thinking: I believe it does @sam?

This is already default on if you are using CJK locales

It was not enabled.
A while ago I made it enabled, some Kanji work but kanatana like トークン still doesn’t get the result.

You probably need to rebuild your index, if you edit a post with the kanatana does it start working?

I rebuild the app but still the same.
Is Index rebuild different from ./launch rebuild app?

AFAIK you can reindex as follows:

cd /var/discourse
./launcher enter app
rake search:reindex

@Stephen
Thank you! I stored backup just in case, and anything else should I do before the reindex?
I don’t think there is any, but I should be careful.
Would be appreciated if you could advise me.

I just did this and found that we can search Kanji. But searching Katakana still does’t find words/topics. Any suggestions could I get?

This problem seems to be unresolved in Version 2.6. The search box used when customizing from English to Japanese is working properly. Even if you include katakana, hiragana, and kanji. Is the search engine different? Is it difficult to solve this problem?

Can you include

  1. A sample paragraph here in katakana

  2. A sample search term that you have that is not working

  3. Confirmation that your site locale is in Japanese or that search tokenize chinese japanese korean is enabled