No search results are shown for stopwords

Product Support
1

Hi friends, I discovered very strange problems on my website.

My website is an online dictionary that contains many words, so every topic is in the format of “WORD meaning-Pronunciation-Example”.

However, when I searched some words in my website, many of them don’t show result.For example, “off”, “few”, “able”, “add”, “age”, “but”, “gain”, “she”, “here”, “him”, “top”, “very”, “why” etc.

And for some other words, when I search the whole word, no result there; but when I search part of the word, the result appears. For example:

when I search “city”, no result; but when I search “cit”, the result appears;
when I search “policy”, no result; but when I search “poli”, the result appears;
when I search “industry”, no result; but when I search “indu”, the result appears;
when I search “should”, no result; but when I search “shou”, the result appears;
when I search “story”, no result; but when I search “sto”, the result appears;
when I search “memory”, no result; but when I search “memo”, the result appears;
when I search “pretty”, no result; but when I search “prett”, the result appears;
when I search “happy”, no result; but when I search “happ”, the result appears;

There are totally about 200 words that have such issue and I believe, if I don’t do something, I will see more and more and more in the future.

So please, my friends, I NEED YOUR HELP :scream_cat:

2

Pasted image880×1067 102 KB

Are you using our official Docker install?

3

@sam Hi Sam, I mean when I search these words in my OWN website, they have the problem.

For example:

Pasted image325×102 5.11 KB

Pasted image347×112 5.43 KB

But these topics DO exist. So I’m wondering why :worried:

4

Those are “stop” words

Stop words are words that are very common, appear in almost every document, and have no discrimination value. Therefore, they can be ignored in the context of full text searching. For example, every English text contains words like a and the, so it is useless to store them in an index. However, stop words do affect the positions in tsvector, which in turn affect ranking:

5

@Mittineague Thank you for your kind reply :slight_smile:

Is that possible to solve the problem? Since my website is an online dictionary, it would be super complicated if my user cannot search the English words they want to study

6

no, it is impossible, there is no setting to disable stemming if we would it would cause severe performance issues for search.

7

@sam :sob: can’t be worse if so… but anyway thank you for your help :+1:

8

Stemming and stop words are unrelated, though. @Jiaqi, I’m pretty sure you can do this, if you’re willing to rummage around enough, but it’ll involve learning a lot more about full-text search and PostgreSQL internals than you ever cared to know. It’s certainly not something that would ever be a “standard” feature, because it’s so incredibly niche. You’d be better off creating some sort of custom index plugin that kept a mapping of all the words and their associated topic IDs, and offered a custom search box to find the relevant topic.

9

There sure is a lot more to it than I ever imagined.

The custom index could be doable. AFAIK there are 127 stops. A bit, but not overwhelmingly so.

https://apt-browse.org/browse/ubuntu/trusty/main/i386/postgresql-9.3/9.3.4-1/file/usr/share/postgresql/9.3/tsearch_data/english.stop

10

The trick, if you want to use a different stop word list (or not have any at all) is to define a custom dictionary and language profile (or whatever PostgreSQL calls it) that doesn’t have the stop words in it, and then reconfigure your full-text indexing to use that instead of the standard English one. I’ve done it once, a long long time ago, and I have no interest in repeating the experience.

12

스톱 워드로 이루어져 있지만 유용한 조합을 이룬 "to do" 같은 구를 인덱스에 포함하도록 강제하는 방법이 있을까요? 이 경우 ToDo를 사용할 수는 있지만, 약간 번거롭습니다. (현재 사이트가 비공개 상태이므로 Google은 선택지가 아닙니다.) 인덱스가 단어만 인덱싱한다면, 이는 어려울 것 같습니다. (이 경우 프론트엔드가 최종 구 파싱을 수행한다고 가정합니다.)

13

기술적으로는 가능하지만, 저희가 계획하고 있는 사항은 아닙니다.

검색 색인 방식을 복잡하게 만들고, 엣지 케이스를 증가시키며, 검색 결과를 설명하기 어렵게 만들 것이기 때문입니다.

14

알겠습니다: 현재 검색 동작은 어디에서 완전히 설명되어 있나요?

15

가장 좋은 확인 방법은 소스 코드, 특히 테스트 스위트와 온라인에 있는 PostgreSQL 전체 텍스트 문서입니다.