Chinese search excerpts appear broken

tgxworld · December 8, 2020, 1:22am

The stop words are determined using GitHub - yanyiwu/cppjieba: "结巴"中文分词的C++版本 · GitHub. With the recent changes to how search excerpts are displayed, we should just remove the following line since it messes with the actual search data.

github.com/discourse/discourse

lib/search.rb

1cf057fb1


      
          # For Japanese we should investigate using kakasi
          if ['zh_TW', 'zh_CN', 'ja'].include?(SiteSetting.default_locale) || SiteSetting.search_tokenize_chinese_japanese_korean
            require 'cppjieba_rb' unless defined? CppjiebaRb
            mode = (purpose == :query ? :query : :mix)
            data = CppjiebaRb.segment(search_data, mode: mode)
          
            # TODO: we still want to tokenize here but the current stopword list is too wide
            # in cppjieba leading to words such as volume to be skipped. PG already has an English
            # stopword list so use that vs relying on cppjieba
            if ts_config != 'english'
              data = CppjiebaRb.filter_stop_word(data)
            else
              data = data.filter { |s| s.present? }
            end
          
            data = data.join(' ')
          
          else
            data.squish!
          end

Either way, our search support for Chinese is not great but there are PG extensions that we may want to consider so that we can properly support languages that do not have native support. Perhaps https://pgroonga.github.io/?

Topic		Replies	Views
Chinese searching is broken Bug	3	1099	October 12, 2016
Searching Chinese terms in middle of sentence Feature	23	3846	October 8, 2016
Chinese search doesn't work to some words Product Support	14	1883	December 14, 2018
What's the word tokenizer for different languages in discourse? Product Support	1	673	May 27, 2020
Korean words can't be searched Product Support	34	2026	May 27, 2018

Chinese search excerpts appear broken

Related topics