CJKV doesn’t have word boundaries. It’s more reliable to use the feature with a sentence level. In a word, support this feature without word boundaries.
Suggested: (Chinese) https://meta.discoursecn.org/t/topic/2175?u=fantasticfears
AI-generated summary
The discussion revolves around improving Discourse’s word censoring feature to better support Chinese (CJKV) text, which lacks clear word boundaries. fantasticfears suggests implementing sentence-level censoring as a more reliable approach. schungx notes that the current system uses \b word boundaries in regex, which fails for non-English languages, and proposes either removing them by default or adding a site setting to toggle them. While a custom build can manually remove \b, schungx admits lacking the Ruby expertise to write a plugin or submit a tested PR. Stranik argues that simply removing \b isn’t enough — the logic file needs full rewriting, and shares a working version using loops. Later, schungx reports the issue is resolved via enabling “watched words regular expressions” in settings, allowing users to manually add \b for English words while freely matching Chinese patterns. Finally, zogstrip confirms that proper Chinese word boundary handling is now available thanks to PR #37844. The consensus leans toward configurable, regex-based solutions rather than hardcoded word boundaries.
1 Like
There is a discussion here:
4 Likes
If you can do a custom build of discourse, it is a simple matter to change that one line of code to remove the wrapping \b’s.
In the long term, I suggest removing them as default, or at least add a site setting for those of us running non-English forums.
1 Like
You could create a plugin to do that and/or submit a PR.
Unfortunately a plugin requires quite a bit of Ruby knowledge. I can debug, but probably not even close to writing plugins.
A PR would require that I fork the entire repo, which is ok except I have no way to test it. It is bad form to submit a PR without testing…
1 Like
There’s really not enough to remove one line. It is necessary to completely rewrite the logic file. I gave there a working version of the file (using loops).
Well, not to remove the line, but to remove the \b’s in the line.
Regexp will never work for all languages with word breaks. The best you can do is to allow the user to decide which words require word breaks and which do not.
With the \b wrapper hard-coded in right now, there is no choice.
This issue is now solved by:
To match Chinese patterns, turn on Settings > Posting > watched words regular expressions.
Beware, your Watched Words will now be raw regular expressions, so if your list includes English words, you’ll need to put in your own word break \b where necessary.
6 Likes
This topic was automatically closed after 40 hours. New replies are no longer allowed.
We should have proper “word boundaries” handling for Chinese now thanks to