Search a term in Japanese

Support
30

I’ve tried tiny_segmenter from Rubygems and at least it does generate the words I’ve listed in the previous comment.

# coding: utf-8
require 'tiny_segmenter'
require 'pp'

s = File.read('topic27.txt')

ts = TinySegmenter.new
sg = ts.segment(s, ignore_punctuation: true)
pp(sg)

bundle exec ruby test.rb | grep -e 北側 -e 真上 -e 一般
 "北側",
 "真上",
 "一般",
 "一般",
 "一般",
 "北側",
 "一般",

A quick search about TinySegmenter told me that the model it uses isn’t as good. There is model generator for it.

I haven’t tried it though.

3 Likes