Thai language support for searching

With CJK languages, each character is (more or less) a word. But in Thai several characters combine to produce a syllable, where the vowel may be before, after, above, or below, the consonant, and various tone markers may also be added. To make it even funner, words are usually (but not always) one syllable and there is usually no space between words. There must be standard libraries to tokenize these for search, I’m guessing.

Incidentally, most of the south and south-east Asian languages use related scripts, so if Thai has these problems we may find other languages do too.

3 likes