This is a useful moderation tool, but it has poor support for non-English characters, and the presence of non-English characters can even affect the detection of English and numbers. Here, taking the Simplified Chinese word “测试” (Test) as an example, the Watched words list contains three elements: “测试”, “Test”, and “123”. In the test below, none of the three examples triggered Watched words.
I searched within the site and found another similar issue about Censored words: Censored words do not respect word boundaries in non-latin alphabet . It seems that this is a common problem across the entire watch word matching system?
2 Likes
Thanks for the report, this will be fixed by
main ← fix/watched-words-cjk-boundaries
merged 05:04PM - 16 Feb 26 UTC
Watched words failed to match in CJK (Chinese, Japanese, Korean) and other space… less scripts because word boundary detection relied on whitespace or non-word characters. Languages like Chinese don't use spaces between words, so "测试" inside "这是一个测试文本" was never matched.
Introduce a SPACELESS_SCRIPTS constant covering Han, Hiragana, Katakana, Hangul, Thai, Lao, Myanmar, Khmer, and Tibetan Unicode ranges. Update `match_word_regexp` for both Ruby and JS engines so that characters from these scripts are treated as word boundaries. This allows a CJK watched word to match when surrounded by other CJK characters, and a Latin watched word to match when adjacent to CJK text (e.g., "Test" in "我的Test很好"), while still preventing partial Latin matches (e.g., "Testing" does not match "Test").
Also fix the admin watched word testing modal to use `RegExp.exec()` with capture group extraction instead of `String.match()`, since the new boundary patterns include a leading consuming group.
Remove the outdated "non-chrome browsers do not support lookbehind" comment — all major browsers have supported lookbehind since 2023.
https://meta.discourse.org/t/71288
https://meta.discourse.org/t/396109
2 Likes
