Chinese search excerpts appear broken

sam · May 20, 2021, 5:50am

I thought I made some changes here:

lib/search.rb

626b8465b


      
          def self.segment_cjk?
            ['zh_TW', 'zh_CN', 'ja'].include?(SiteSetting.default_locale) ||
              SiteSetting.search_tokenize_chinese_japanese_korean
          end

Is your locale set to zh_TW , zh_CN or ja? if not is search_tokenize_chinese_japanese_korean set to true?

We have a bypass here:

github.com/discourse/discourse

lib/search/grouped_search_results.rb

626b8465b


      
          if post.post_search_data.version > SearchIndexer::MIN_POST_REINDEX_VERSION && !Search.segment_cjk?
            if SiteSetting.use_pg_headlines_for_excerpt
              scrubbed_headline = post.headline.gsub(SCRUB_HEADLINE_REGEXP, '\1')
              prefix_omission = scrubbed_headline.start_with?(post.leading_raw_data) ? '' : OMISSION
              postfix_omission = scrubbed_headline.end_with?(post.trailing_raw_data) ? '' : OMISSION
              return "#{prefix_omission}#{post.headline}#{postfix_omission}"
            else
              opts[:cooked] = post.post_search_data.raw_data
              opts[:scrub] = false
            end
          else
            opts[:cooked] = post.cooked
          end
          
          GroupedSearchResults.blurb_for(**opts)

Topic		Replies	Views
Chinese searching is broken Bug	3	1099	October 12, 2016
Searching Chinese terms in middle of sentence Feature	23	3846	October 8, 2016
Chinese search doesn't work to some words Product Support	14	1883	December 14, 2018
What's the word tokenizer for different languages in discourse? Product Support	1	674	May 27, 2020
Korean words can't be searched Product Support	34	2027	May 27, 2018

Chinese search excerpts appear broken

Related topics