Chinese search issues


(Leung Aaron) #1

Now the Chinese search function fails to work completely. I had tested from a previous built from Bitnami, and the search works well, but now in the 0.9.8.5 and even the 0.9.8.7 version, it is completely broken , and I could just only search by english keywords.

There is also one issue that I am very concerned about : Will there be compatibility problems if I backup from an older version, than restore to a newer version? I tried to restore from a 0.9.8.5 backup file after upgraded to 0.9.8.7 , but it seems to fail to restore successfully.

Please have a look in these matters, thanks!


Chinese search failed
Search function for Korean text works only new topic after upgrade
(Leung Aaron) #2

I would like to explain a bit further on the search issue :

for example ,in meta.discourse.org there are also some topics that contains chinese characters, and we can use them as a search experiment.

Let’s use this one :

Let’s try to use some chinese characters inside this topic as keywords to perform some search queries: (assuming now the search minimum limit is 3 characters as default)

(1) 官方推 <— SUCCESS

(2) 方推出 <— FAIL

(3) 本指南 <— FAIL

(4)为此, <— SUCCESS

(5) 。为此 <— SUCESS

According to this small experiment , it seems that only a Chinese keyword that is following after a punctuation mark can be searched, no matter which locale the discourse forum is using.

This is the case in meta.discourse.org .
However in my own website (http://love-liver.net) , no matter how I search for Chinese or Japanese Characters, no result will show up, and only English keywords can be searched.

My method used for installing Discourse : Latest Recommended Docker in DigitalOcean Method

(PS. discoursehosting.net and bitnami builds of discourse can be able to search , but it seems that they are not using the same updated version , up to now is 0.9.8.8 )


(Larry Salibra) #3

The version running on Meta also seems to have the same problem with Chinese search.

With regards to the missing translations, I imagine you can fork discourse, fix the translations and send a pull request on github. Not sure if there’s someone in charge of particular languages.


(Leung Aaron) #4

It would be my pleasure to help and contribute with the Discourse project, however I am not familiar with the codings, and I am not sure where should I edit to change the corresponding fields. (For the favourite / starred part, I had made a pull request on github already, but for the other parts I have no clue how to fix them )
https://github.com/discourse/discourse/pull/2035


(Jeff Atwood) #5

You are mixing two different issues here. For localization see

For search, you must tell Postgres to use the appropriate language for the database indexes.


(Leung Aaron) #6

sorry for mixing the issues, maybe moderators can help and splitting the issues to 2.

I am a website administrator and not really good in programming, and I have almost no knowledge in postgres. So in a docker-digitalocean installation, how can I " tell Postgres to use the appropriate language for the database indexes" ? ( It would be quite easy in a phpmyadmin evrionment, but I don’t know what it will be in a postgres case )

I believe this is a big issue for every non-English non-Latin Discourse users.


(Leung Aaron) #7

Also , I had searched the past topics, for example ,this one :
https://meta.discourse.org/t/can-t-search-with-chinese-keywords/4500?source_topic_id=13287

It seems that the older version codes have no problem in searching Chinese.

In fact, in the trial period using discoursehosting.com services and my localhost Bitnami build , both seems to be able to search without problems too. ( But their versions are definitely not the updated latest version)

So I guess there should be something to do with the codes itself, but not some external settings.


(Leung Aaron) #8

So are there any solutions up to now ?
This will be a epic miss and failure if Discourse fails to search with languages other than English when it releases as version 1.0 .


(Jeff Atwood) #9

As mentioned before, you must tell Postgres to use the correct languages for its database search indexes. Refer to the Postgres documentation for specifics.


(Leung Aaron) #10

I am not a programmer , so I have no idea how to do so ( copy / pasting the codes in the tutorial to install is my best already ) , and I believe many end users also won’t have this ability to change these settings manually.

At least there should be a well documented tutorial for how to change the specific settings ?
(As a matter of being “The wordpress of forums” , I do believe this should be a no-hassle automated process though )


(Dmitry) #11

Agree, even I, as a developer, can’t find good documentation of how is to work with fulltext indexes in the PostgreSQL. That one not helped: PostgreSQL: Documentation: 9.1: Tables and Indexes

I’ve made issue on a github: https://github.com/discourse/discourse/issues/1956


(Leung Aaron) #12

So it seems that not just Chinese, I guess Japanese , Korean , Russian… all non-Latin languages will have the same search issues…


(Leung Aaron) #13

So weeks has passed. Still no progress ?
Any developers can tell me what’s going on ?


(Leung Aaron) #14

several new versions released, still failed to search Chinese & Japanese.
Dear @codinghorror , is it going to have a flawed search function even in version 1.0 ?

I just hope someone can tell me whether this is going to be officially solved, or nobody is going to take care of this matter.


(Dmitry) #15

@aaroleung it can be solved with current version. Though I didn’t had enough time to work on it and present some solution.


(Leung Aaron) #16

Any clues or advices that can be given so that I could try it out for myself ?


(Dmitry) #17

@aaroleung I guess it was the following algorithm: backup database data, create correct system charset in ubuntu, then create database schema again, update indexes to correct ones, and add data. But I cannot say it’s all true. That’s what I read how are other updating the index with a correct collocation.

Partially it answered here:


(Leung Aaron) #18

is it possible for me to change the postgresql settings even when I used the docker installation method ?


(Leung Aaron) #19

as docker install method finished everything as default , I have no idea what the name of postgresql database is, not the admin user name and pass used for the postgresql system…


(Sam Saffron) #20

try: sudo -iu postgres