English words can be searched properly
but the forum doesn’t recognize Korean and Chinese search terms at all
AI-generated summary
The discussion revolves around the issue of Korean words not being searchable on a Discourse forum. k11 reports that English words are searchable, but not Korean or Chinese. The problem is investigated, and codinghorror suggests checking the database locale and language, which k11 confirms is set to Korean.
k11 is advised to enable the “search tokenize chinese japanese korean” feature, but this does not solve the issue. schungx suggests that recooking posts may be necessary to update the full-text index.
Further investigation reveals that the problem is due to the lack of word splitting in Korean search terms. sam explains that search relies on having separate words in the index, but Korean words do not have spaces between them. k11 disputes this, saying that the issue is not about spaces.
sam suggests changing the “min search term length” to 2 or 1, which k11 finds solves the problem. However, this also causes irrelevant search results to appear.
k11 discovers that the search query is being split into individual syllables, even though the search term does not have spaces. This is why the search results are not showing up.
sam provides a commit that removes word segmentation from Korean and changes the min search length to 2 for Korean and Japanese. k11 is advised to try this out.
Did you install using the correct database locale / language?
yes I did. I installed as Korean language.
should I turn on ‘search tokenize chinese japanese korean’ this feature??
I had someone contact me about this recently. I wasn’t sure which locale to use for Chinese, as I don’t know how the Postgres locales map to the various Chinese dialects. Is there some way to know which to use?
should I turn on ‘search tokenize chinese japanese korean’ ???
please help me.
Probably, give it a shot and see what happens.
turned on it but still can’t search any Korean and Chinese words..
I don’t know for certain, but I have a feeling the tokenize happens between raw → cooked.
If you rebake a post with the Korean or Chinese words, does that post then show up in Search?
I posted a new topic and searched korean words but still nothing shown up in search result 
Not sure, maybe @fantasticfears would know?
You don’t have to change Postgres locale but your default_locale should be zh_CN and zh_TW. @k11 you could try search tokenize chinese japanese korean but the dictionaries are limited for Korean.
I already turned on ‘search tokenize chinese japanese korean’ like 12 hours ago
but site cannot search any korean words..
Should I reboot my server or do something else?
Dont you have to recook your posts to update the full text index?
If i remember, getting Chinese working is nontrivial. There are a whole bunch of settings required for CJK support. For example, you’ll need to turn down the text entropy limit, and turn off prettify.
can you help me setting Korean also?
still couldn’t solve this issue
can’t search any korean words..
anyone can help?
Try creating a new post, does it show up in search?
new post also not show up in search result..
only english can..
Specific tokenizers are needed for Japanese and Korean to work. (Maybe Korean doesn’t need one?)
It seems that Discourse relies on spaces between words to perform a success search. Chinese and Japanese users don’t add a space between two words, therefore a tokenizer is required to split a sentence into words. Though I don’t understand Korean (and Japanese), there are inter-word spaces on Korean Wikipedia, however.
CppJiebaRb, the tokenizer designed for Chinese only, has been used for Chinese, Japanese and Korean in Discourse. It segments any Japanese or Korean sentence into an array of each character but not words which are expected.
thanks for comment.
But what should I do for korean search?
sorry I’m very new to SSH command and not sure what you meant me to do..
Nice detective work, looks like it is a bit complicated:
@k11 the problem is search relies on having seperate words in the index.
So for example if we do not try to inject spaces when we type 공을 치다 you will not be able to find 을.
Curious when you use search on this site does search in work correctly for this topic?