Search results not found using web GUI but found in command line console


(Marco) #1

Hello there!
I’m in the process of porting my old smf2 forum to Discourse, and among the tests we run to ascertain that the process is complete and thorough we run some search tests on both platforms.

A couple of days ago one of our “beta testers” found out a weird behavior: searching a specific string (we’re Italians but the string has no special nor accented characters in it) such as dove sono tutti quanti leads to no results whatsoever in Discourse using the search function of the website, although we know we have at least 3 posts containing this string, and we can successfully find those in smf2. See pictures below here.

Inside Discourse I can actually locate and get to the posts, proving that they have been correctly converted and imported from smf2, and the very string I’m searching is right there (bottom right part of the text).

I then run the very same query using the command line console of Discourse, and this time all the posts I have on smf2 are found!

discourse_console1

So, it seems to me something happens in the forum GUI front-end when it is supposed to display the search results in the web interface.
Am I doing something wrong, is it a bug, or is it something related to having some quote or double quote in the raw text of the posts?


(Régis Hanol) #2

Search is a tiny bit more complicated than a where clause on raw text :wink:

Have a look at the search.rb file.


(Marco) #3

I’m sure of that. :wink:
My question can have this TLDR version, then: how is it possible that a simple query (simple as there is no further search criteria specified) submitted in the GUI returns zero results, wheras the very same query run from command line returns exactly the posts I expected to be found?

My “I don’t know better” hypotesis is that something brakes the return of the found posts from the DB up to the web GUI level. I might of course be wrong.


(Mittineague) #4

My guess is that those are “stop” words that Postgres considers unhelpful for text searches.

If google translate is correct, “where everyone is” are all commonly used words.

In other words what you are seeing is sort of like Postgres saying “I don’t have anything good in that input to use in a search”, while the command line says “I’ll look for that sequence of bytes”.

I wouldn’t expect a Postgres search to work with stop words, but it would be nice if surrounding the stop words with quotation marks forced the issue. IIRC, this has been discussed before and is how a search now works, but maybe not. eg.

[dove sono tutti quanti] - won’t work
["dove sono tutti quanti"] - may work


(Marco) #5

Correct.

Negative, I tried. And the point is, these are italian words, which when used by themselves in a search are bringing up other posts…
And again, the search is successful if done on the Ruby command line…


(Mittineague) #6

Postgres and Ruby searches are different. Ruby searches are more thorough but are less efficient. Postgres searches reduce words to lexemes and though not as thorough are much more efficient in terms of resource use and time to result. It’s a trade off that works well for “significant” words but fails for “ubiquitous” words.