# What's the word tokenizer for different languages in discourse?
**URL:** https://meta.discourse.org/t/whats-the-word-tokenizer-for-different-languages-in-discourse/152893
**Category:** Support
**Created:** [May 27, 2020, 12:53am UTC](https://meta.discourse.org/t/whats-the-word-tokenizer-for-different-languages-in-discourse/152893 "2020-05-27T00:53:42Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@xiasummer](https://meta.discourse.org/u/xiasummer)
#### Post date: [May 27, 2020, 12:53am UTC](https://meta.discourse.org/t/whats-the-word-tokenizer-for-different-languages-in-discourse/152893/1 "2020-05-27T00:53:42Z")
Hi, I was using discourse in Chinese. And I found that many words are not tokenized well.
In a sentence, we have many words, and we need to separate them well, to support keyword searching and other important functions.
While I have to say, I believe that the Chinese words tokenizer of the discourse works not good enough.
Is that an “Old” tokenizer? Can we replace it with new ones?
* * *
If you can read Chinese words, here are my findings:
[https://meta.discoursecn.org/t/topic/2414](https://meta.discoursecn.org/t/topic/2414)
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [May 27, 2020, 1:07am UTC](https://meta.discourse.org/t/whats-the-word-tokenizer-for-different-languages-in-discourse/152893/2 "2020-05-27T01:07:50Z")
We use [GitHub - erickguan/cppjieba\_rb: Cppjieba Ruby binding · GitHub](https://github.com/erickguan/cppjieba_rb)
Which is based on [GitHub - yanyiwu/cppjieba: "结巴"中文分词的C++版本 · GitHub](https://github.com/yanyiwu/cppjieba)
@fantasticfears built the gem that enables Ruby to have support for this.
Are you noticing any specific issues you would like addressed?