# Searching Chinese terms in middle of sentence

**URL:** <https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440>\
**Category:** Feature\
**Created:** [Novembro 10, 2015, 11:27pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440 "2015-11-10T23:27:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dleung](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dleung](https://meta.discourse.org/u/dleung)\
**Post date:** [Novembro 10, 2015, 11:27pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/1 "2015-11-10T23:27:41Z")

</div>

When search using for Chinese keywords, any terms in the middle of the sentence do not turn up in search results. Searching on the first few words of a sentence however works.

Here’s a topic:

 ![](https://global.discourse-cdn.com/meta/original/3X/1/f/1f0400fa68e288a9f88cc2c19e3fbc773d17129c.png)

Searching for terms in middle of sentence doesn’t work:  
 ![](https://global.discourse-cdn.com/meta/original/3X/b/2/b25be8d4963c643a2c75e174ac1e97ed93cc0944.png)

Including the first few words in the beginning of the sentence works:

 ![](https://global.discourse-cdn.com/meta/original/3X/6/c/6cf3dfa5598d104499153d02c9ef1c7b9740ceb0.png)

This phenomenon is mentioned in the first half of this thread:  
[https://meta.discourse.org/t/can-t-search-with-chinese-keywords/4500/10](https://meta.discourse.org/t/can-t-search-with-chinese-keywords/4500/10)

It suggests that we need to turn on site wide Chinese locale in order to enable such search. Is this still the case?  
We’re currently on En locale but need to support multiple languages.

Thanks.

---

<div class="post-metadata">

**Author:** ![dleung](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dleung](https://meta.discourse.org/u/dleung)\
**Post date:** [Novembro 10, 2015, 11:41pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/2 "2015-11-10T23:41:55Z")

</div>

Actually I think this might be because “量子阱” is a rare term and not recognized as a keyword?  
Anyway, is there any viable way we can improve this?

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Novembro 11, 2015, 1:22am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/3 "2015-11-11T01:22:15Z")

</div>

Can you post the exact terms/sentence being used, I can enable a mixed search tokenizer behind a site setting

---

<div class="post-metadata">

**Author:** ![dleung](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dleung](https://meta.discourse.org/u/dleung)\
**Post date:** [Novembro 11, 2015, 1:55am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/4 "2015-11-11T01:55:12Z")

</div>

Are there any plugins that can include rare translated terms like that automatically?  
There are many such terms that is specific to our field in photonics but we don’t keep track of it 'cause there are many.  
Or we can try to find such a glossary list on the internet and send to you in some format? but it might be a very time-consuming exercise for us.

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Novembro 11, 2015, 2:11am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/5 "2015-11-11T02:11:02Z")

</div>

We have a tokenizer that is enabled for Chinese locale I can add a site setting to enable it unconditionally

Just need some examples pasted here of sentences and expected word breaks so I can confirm it works well, can not use the pics need it in text

---

<div class="post-metadata">

**Author:** ![dleung](https://avatars.discourse-cdn.com/v4/letter/d/a183cd/32.png) [@dleung](https://meta.discourse.org/u/dleung)\
**Post date:** [Novembro 11, 2015, 5:18pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/6 "2015-11-11T17:18:42Z")

</div>

I see. Here’re the text for the example:  
Topic: 能否设计量子阱材料  
search terms: 量子阱, 设计量子阱, 量子阱材料, 材料, 设计  
Is it possible to enable it for Korean and Japanese as well?  
thanks!

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Novembro 27, 2015, 5:38am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/7 "2015-11-27T05:38:27Z")

</div>

This is now completed per:

[https://github.com/discourse/discourse/commit/f74a6457ee9a3ef4cb9f0d4283a8604e5b05551e](https://github.com/discourse/discourse/commit/f74a6457ee9a3ef4cb9f0d4283a8604e5b05551e)

We will have it deployed on the business tier next week. The site setting `search_tokenize_chinese_japanese_korean` will enable the CJK tokenizer for search regardless of locale.

To take effect you will have to enable the site setting and edit the topic in question (to refresh the search index)

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 2, 2016, 11:21am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/8 "2016-10-02T11:21:18Z")

</div>

> [@sam](#):
>
> To take effect you will have to enable the site setting and edit the topic in question (to refresh the search index)

Hi, am I reading this right? Does this mean the only way to ensure that old posts turn up in such searches is by editing them individually? Thanks.

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Outubro 2, 2016, 12:33pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/9 "2016-10-02T12:33:51Z")

</div>

Once enabled we can kick off another reindex for you

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 5:11am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/10 "2016-10-05T05:11:25Z")

</div>

Hello, that has been done, but some unusual problems seem to be occurring.

Searches for many Chinese terms which appear on the website are not returning any result. One thing I am noticing is that problems seem more likely to occur in searches for terms including traditional characters (as are used in Taiwan) for which there are simplified (as used in China) equivalents. In some case I don’t get results if the term includes one such character, and less then three characters for which there is no such equivalent (the character is the same in Taiwan and China)

Some things I’m seeing: 台北 (Taipei) gets plenty of hits, but 台灣 (Taiwan) doesn’t get any (I changed our minimum search length setting to 2 to accommodate such Chinese place names/terms.) (characters with simplified equivalents bolded).

**顆** 老鼠屎壞了一鍋湯 gets results, but some smaller subsets including a bolded character don’t. A lot of common terms like the name of President Lee I mentioned,李登輝 aren’t getting any results at all, even the test post I made a few days ago (which was being found previously.) Even a “search this category” search in that category or thread do not find the post. The same is true of 李登, which would seem to contradict my traditional/simplified theory. Perhaps I’m barking up the wrong tree with that, I don’t know.

Something else odd seems to be happening in English searches, that I did not notice previously. For example, a search on “anyone who I know” (no quotes) is returning a lot of results for “I”, even in words where the letter is present. I’m not sure if this would be expected or not.

I hope this makes some sense. Thanks for the assistance.

---

<div class="post-metadata">

**Author:** ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)\
**Post date:** [Outubro 5, 2016, 5:34am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/11 "2016-10-05T05:34:19Z")

</div>

My first guess is that the “dictionary” being used for Postgres searches is omitting what would be Chinese "stop words\* (words that are so common they are considered to be of little to no value for searches).

But I do not know where or how Discourse configures what dictionary to use based on the locale to support or discredit the guess. .

I don’t understand Chinese but maybe you can see something here?

> **[Google Code Archive - Long-term storage for Google Code Project Hosting.](https://code.google.com/archive/p/nlpbamboo/wikis/TSearch2.wiki)**

I find the documentation fairly bewildering even when it is in English, that I can read, but hopefully you find it easier going.

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 5:54am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/12 "2016-10-05T05:54:40Z")

</div>

> [@Mittineague](#):
>
> My first guess is that the “dictionary” being used for Postgres searches is omitting what would be Chinese "stop words\* (words that are so common they are considered to be of little to no value for searches).

I would guess that wouldn’t be a concept that could be applied to Chinese searches. In a nutshell, there aren’t characters that are common enough in the way that “the” for example is common, that it would make searching for them unprofitable, and it would make it impossible to search for any terms including that character.

> [@Mittineague](#):
>
> I find the documentation fairly bewildering even when it is in English, that I can read, but hopefully you find it easier going.

I don’t 🙂 It seems to say that PostgreSQL tokenized searches require the installation of Bamboo, and some instructions for doing so.

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Outubro 5, 2016, 6:01am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/13 "2016-10-05T06:01:03Z")

</div>

> [@tempogain](#):
>
> (Taiwan) doesn’t get any

Can I get a couple of sentences in context so I can test the word splitter?

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 6:38am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/14 "2016-10-05T06:38:46Z")

</div>

> [@tempogain](#):
>
> 台灣

Sure, here are some with the characters for Taiwan, let me know if you need other examples:

台灣籍

> **[嫁娶外籍配偶的結婚手續 - Getting Married - for your TW spouse to read](https://tw.forumosa.com/t/getting-married-for-your-tw-spouse-to-read/40819)**
>
> 公証結婚 外籍: 單身証明 到所屬國家的在台辦事處辦理，並送至外交部領事事務局做文件驗証 boca.gov.tw/ct.asp?xItem=1284&ctNode=16&mp=1 在做文件驗証時，外交部會把正本收走，只給你影本，所以最好多申請1-2份，以免弄不見就又要再跑一趟‧ 台灣籍: 身分證正本、印章及具有台灣公民身份的証人2位 (20歲以上，帶身分證正本及印章)‧沒有印章簽名也可以的！ 結婚登記 外籍: 護照、中文姓名聲明書 ...

但我爸爸是台灣公民

> **[ABC obtaining Taiwanese Passport](https://tw.forumosa.com/t/abc-obtaining-taiwanese-passport/86359)**
>
> 如果別人已經問過我表的主題請告訴我！給我鏈接～ 我是在美國出生的，但我爸爸是台灣公民。我好像在哪裡讀過如果你爸媽是出生時是台灣公民，公民入籍就能流傳下來給小孩。對嗎？我為了工作機會及看親屬很想拿台灣公民入籍但是現在是在不知道該怎麼辦。好像小孩超過十八歲就拿不到ROC ID卡。再過幾個禮拜就是我二十一歲的生日… 所以可能太晚了。挫賽！ 😢 😢 有人知道超過幾歲就不能拿ROC ID卡嗎？ :ponder:...

台灣沙發衝浪客交流

> **[Taiwan CouchSurfing 台灣沙發衝浪客交流 Facebook group](https://tw.forumosa.com/t/taiwan-couchsurfing-facebook-group/72627)**
>
> Join our Taiwan CouchSurfing 台灣沙發衝浪客交流 Facebook group, facebook.com/groups/taiwan.couchsurfing/ !

台灣是中國的一部份

> **["台灣是中國的一部份" translates Google from English "Taiwan is not a part of China"](https://tw.forumosa.com/t/translates-google-from-english-taiwan-is-not-a-part-of-china/74160)**
>
> Try it yourself: translate.google.de/#en/zh-TW/ta … of%20china The only way to get around it is to write “not” in capitals, which is what I would suggest my Taiwanese friends to always do in the future.

紐約時報曾形容台灣對亞洲同志來說

> **[【Taipei LGBT and the City Tour 同志台北】2016/01/31](https://tw.forumosa.com/t/taipei-lgbt-and-the-city-tour-2016-01-31/88945)**
>
> 【Taipei Theme Tour｜LGBT and the City 同志台北】 The New York Times once described Taiwan as the beacon with inspiration for Asia’s gays. But do you know what were the gay lives like in the dark past? What are the steps taken to learn to embrace...

但我的台灣護照已過期一年多了

> **[Can I enter Taiwan with an expired Taiwan Passport?](https://tw.forumosa.com/t/can-i-enter-taiwan-with-an-expired-taiwan-passport/76150)**
>
> I was born in Taiwan but immigrated to Canada since I was a kid. I have both Taiwanese + Canadian passport, however my Taiwan passport has expired since over an year ago. I currently live in Terrebonne, Quebec, Canada and it’s very inconvenient to...

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Outubro 5, 2016, 6:41am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/15 "2016-10-05T06:41:37Z")

</div>

Can you make it a bit easier for me, 3 strings of say 20 chars or so that exhibit the problem?

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 6:45am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/16 "2016-10-05T06:45:54Z")

</div>

Sure. I’m starting work now, but I will test for a few such strings and post back later.

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 9:37am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/17 "2016-10-05T09:37:17Z")

</div>

This string doesn’t find the first post above (Getting Married):

國外健檢證明須經中華民國駐外館處驗證

I’ve found one so far. It seems hard to find such long strings that fit the bill–most work. I’m still looking.

This string 亦應視同受承辦檢察官所選 任或囑託而執行鑑定業務 doesn’t find this post (most other strings did)

[http://tw.forumosa.com/t/zain-dean-conviction-fatal-hit-run-case-part-ii/64402](http://tw.forumosa.com/t/zain-dean-conviction-fatal-hit-run-case-part-ii/64402)

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 5, 2016, 3:56pm UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/18 "2016-10-05T15:56:24Z")

</div>

Here’s another:

收元大證券馬志玲兩億元賄賂，協助元大購併復華金

doesn’t find this post:

> **[Ah-Bian didn't pocket a single cent](https://tw.forumosa.com/t/ah-bian-didnt-pocket-a-single-cent/48229/43)**
>
> \[quote=“Hokwongwei”\]Again… Did CSB pocket public money and try to hide it? In my mind, undoubtedly yes. Should he be in prison? Not if the court isn’t able to convict him on reasonable grounds. It’s basically the same as the Dean Zain… Zean Dain…...

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Outubro 6, 2016, 2:46am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/19 "2016-10-06T02:46:22Z")

</div>

I think our lines are crossed a bit so let’s start with an example like this:

Sentence: iliketaiwanalot  
word: taiwan

I am trying to test if our tokenizer that turns: “iliketaiwanalot” to “i like taiwan a lot” is working correctly.

---

<div class="post-metadata">

**Author:** ![tempogain](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tempogain/32/134055_2.png) [@tempogain](https://meta.discourse.org/u/tempogain)\
**Post date:** [Outubro 6, 2016, 4:22am UTC](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440/20 "2016-10-06T04:22:40Z")

</div>

I think so 🙂

I hope I have this right now. 3 sentences, 20 characters or so, including the term Taiwan?

我非常喜歡台灣 is I like Taiwan a lot.

我們的總統目前不敢說台灣是個獨立的國家

Our president currently does not dare to say that Taiwan is an independent country.

Punctuation OK?

台灣主要的城市包括台北、台中、台南、高雄、台東和花蓮

Taiwan’s main cities include Taipei, Taichung, Kaohsiung, Taidong and Hualian

在我認識的人中，大部分不知道台灣在哪裡或以為是泰國

Of the people I know, most don’t know where Taiwan is or think it is Thailand

[Próxima página](https://meta.discourse.org/t/searching-chinese-terms-in-middle-of-sentence/35440.md?page=2)