# Diacritics and search
**URL:** https://meta.discourse.org/t/diacritics-and-search/73744
**Category:** Feature
**Created:** [9 בנובמבר, 2017, 6:57am UTC](https://meta.discourse.org/t/diacritics-and-search/73744 "2017-11-09T06:57:25Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@icaria36](https://meta.discourse.org/u/icaria36)
#### Post date: [9 בנובמבר, 2017, 6:57am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/1 "2017-11-09T06:57:26Z")
Is it possible that Discourse’s search engine doesn’t know about fuzzy search with diacritics?
[Diacritics](https://en.wikipedia.org/wiki/Diacritic) are glyphs added to letters in many languages (notably not in English, except when borrowing foreign words and even then). Speakers of languages with diacritics expect search engines to basically ignore them (i.e. searching for “Mexico” or “México” provides the same results).
In Discourse, diacritics seem to be critical: 😉
- [Redirecting...](https://confederac.io/search?q=poesia) (no diacritic) provides zero results.
- [Redirecting...](https://confederac.io/search?q=poes%C3%ADa) (with diacritic) provides one result.
Languages with diacritics use them everywhere, but then their speakers are often lazy or wrong using them. The difference between finding the content you are looking for or not is huge.
I Discourse using an own search engine or embedding an existing one i.e. Elasticsearch? In any case, hopefully there is a library or something that Discourse developers could use to integrate diacritics fuzziness in search?
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [9 בנובמבר, 2017, 10:09am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/2 "2017-11-09T10:09:21Z")
I remember that the full text search index built works best with English text… There have been a lot of discussions on meta regarding Chinese.
For other languages, you may search meta further. However it may involving changing the default locale of PostgreSql or similar that I remember reading here.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [9 בנובמבר, 2017, 3:27pm UTC](https://meta.discourse.org/t/diacritics-and-search/73744/3 "2017-11-09T15:27:29Z")
If you are talking about non English sites you need to set the database encoding to match the language the site is in.
---
### Author:  [@icaria36](https://meta.discourse.org/u/icaria36)
#### Post date: [9 בנובמבר, 2017, 8:57pm UTC](https://meta.discourse.org/t/diacritics-and-search/73744/4 "2017-11-09T20:57:37Z")
Right, that might explain. Currently the locale is English. The plan is to change to Catalan after resolving [Catalan translation](https://meta.discourse.org/t/catalan-translation/73298).
When the default locale of the site is changed, is the corresponding database encoding automatically changed?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [10 בנובמבר, 2017, 12:18am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/5 "2017-11-10T00:18:26Z")
No, you must set that at install time. How do we set database encoding during setup @techapj? I can’t recall.
---
### Author:  [@MakaryGo](https://meta.discourse.org/u/MakaryGo)
#### Post date: [10 בנובמבר, 2017, 12:22am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/6 "2017-11-10T00:22:25Z")
I believe it has something to do with
```plaintext
db_default_text_search_config: "pg_catalog.english"
```
In the .yml file
---
### Author:  [@icaria36](https://meta.discourse.org/u/icaria36)
#### Post date: [10 בנובמבר, 2017, 7:10am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/7 "2017-11-10T07:10:20Z")
Please bare with me, I don’t know much about databases. If someone has a Discourse instance set to English as default language that has already real users and real content… What do they need to do in order to change the default locale effectively, in addition to changing the `default locale` setting in the admin interface?
A step by step process would be very helpful. Discourse installation documentation is so good that it drove someone like me successfully through the entire process. 😉 The little detail about the implications of defining a locale in a fresh installation could be documented too, in order to avoid surprises like this one.
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [10 בנובמבר, 2017, 11:19am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/8 "2017-11-10T11:19:09Z")
An already-existing database is always a problem regardless what database engine you use… I doubt it is very simple to change database locales, especially when there is already data in it.
I’d suggest you search via Google to see how to change PostgreSQL database default locale when the database is not empty.
---
### Author:  [@icaria36](https://meta.discourse.org/u/icaria36)
#### Post date: [11 בנובמבר, 2017, 9:20am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/9 "2017-11-11T09:20:34Z")
This might be a silly question. Would having the database encoded to UTF-8 by default in Discourse solve these kinds of problems? Isn’t UTF-8 as good for English as for many other languages with diacritics and more?
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [11 בנובמבר, 2017, 11:46am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/10 "2017-11-11T11:46:30Z")
UTF8 is an _encoding_. It does not know of languages and locales and how to split and compare words.
---
### Author:  [@icaria36](https://meta.discourse.org/u/icaria36)
#### Post date: [12 בנובמבר, 2017, 2:37pm UTC](https://meta.discourse.org/t/diacritics-and-search/73744/11 "2017-11-12T14:37:26Z")
> **[How to change the locale of a PostgreSQL cluster](https://makandracards.com/makandra/18643-change-locale-postgresql-cluster)**
>
> There may be reasons to change the locale of your Postgres cluster. A popular one is your development system's locale being used by default (which may be annoying). Here is how to do that. Beware: ...
I think I will live with this problem for now. Anyway my site is multilingual and therefore no single locale will fit perfectly.
Maybe at some point in the future Discourse’s search allows for more fuzziness in general (users have to be accurate in English searches as well). That would probably be good enough to solve the problem of diacritics regardless of language.
---
### Author:  [@MakaryGo](https://meta.discourse.org/u/MakaryGo)
#### Post date: [12 בנובמבר, 2017, 5:24pm UTC](https://meta.discourse.org/t/diacritics-and-search/73744/12 "2017-11-12T17:24:58Z")
Well, _actually_ I found out it would require enabling for given language in the database an extension called ‘_Unaccent_’, but I’m afraid this is somewhat above my expertise with pgsql, sorry.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [1 במאי, 2018, 6:26am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/13 "2018-05-01T06:26:58Z")
I think this is a feature request if anything, but looking at the doco by @MakaryGo I think running something like:
```plaintext
./launcher enter app
sudo postgres psql discourse
discourse=# CREATE EXTENSION unaccent;
discourse=# CREATE TEXT SEARCH CONFIGURATION en ( COPY = english);
discourse=# ALTER TEXT SEARCH CONFIGURATION en
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, english_stem;
discourse=# select to_tsvector('en','Hôtels de la Mer');
to_tsvector
---------------------------------
'de':2 'hotel':1 'la':3 'mer':4
(1 row)
```
I do wonder @codinghorror if we should unaccent out-of-the-box or not.
---
### Author:  [@tophee](https://meta.discourse.org/u/tophee)
#### Post date: [2 במאי, 2018, 5:44am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/14 "2018-05-02T05:44:20Z")
> [@sam](#):
>
> unaccent out-of-the-box or not.
If you decide to unaccent out of the box, it would be good if this could be overwritten by using quotation marks:
poesía =\> poesia
but
“poesía” =\> poesía
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2 במאי, 2018, 5:46am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/15 "2018-05-02T05:46:58Z")
sadly it is not that simple, we only carry one index around and this would impact said index. The only easy change is default unaccent, anything else is major engineering work.
---
### Author:  [@MakaryGo](https://meta.discourse.org/u/MakaryGo)
#### Post date: [6 במאי, 2018, 4:47pm UTC](https://meta.discourse.org/t/diacritics-and-search/73744/16 "2018-05-06T16:47:54Z")
I’ve managed to add Polish fulltext search, thanks to PostgreSQL ability to use hunspell dictionaries for languages that don’t have snowball stemmer algorithm and this Github script:
[https://github.com/lemonskyjwt/plpstgrssearch](https://github.com/lemonskyjwt/plpstgrssearch)
It does unaccent as well and should be easy to use with other languages as well.
I’d submit it as a PR, but it’s not really production quality, rather a hack. But perhaps some :discourse: dev will be able to make it work, what do you think @sam?
It’s a `pups` template that’s later included to main app.yml
Edit:
Unfortunately I’ve managed for it to work only on new install, adding it to existing instance does not seem to work, even after issuing `rake search:reindex`
```pups
params:
LANGFULL: polish
IDENTIFIER: pl_PL
run:
- exec:
cmd:
- apt-get update
- apt-get install postgresql-server-dev-10
- wget https://raw.githubusercontent.com/lemonskyjwt/plpstgrssearch/master/pg_hunspell_install
- chmod +x pg_hunspell_install
- /bin/sh pg_hunspell_install pl PL $LANGFULL
hooks:
after_postgres:
- exec: su postgres -c 'psql template1 -c "create extension if not exists unaccent;"'
- exec: su postgres -c 'psql $db_name -c "create extension if not exists unaccent;"'
- exec:
stdin: |
CREATE TEXT SEARCH DICTIONARY $LANGFULL_hunspell (TEMPLATE = ispell, DictFile = $IDENTIFIER, AffFile = $IDENTIFIER, StopWords = $LANGFULL);\nCOMMENT ON TEXT SEARCH DICTIONARY $LANGFULL_hunspell\nIS '[USER ADDED] Hunspell dictionary for $LANGFULL';\nCREATE TEXT SEARCH CONFIGURATION public.$LANGFULL (COPY = pg_catalog.english);\nALTER TEXT SEARCH CONFIGURATION $LANGFULL\nALTER MAPPING FOR asciiword, asciihword, hword_asciipart, word, hword, hword_part\nWITH $LANGFULL_hunspell, unaccent, simple;\nCOMMENT ON TEXT SEARCH CONFIGURATION $LANGFULL\nIS '[USER ADDED] configuration for $LANGFULL';
cmd: su - postgres -c 'psql discourse'
```
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31 באוגוסט, 2018, 1:25am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/17 "2018-08-31T01:25:49Z")
OK I know this is a newer topic but all the discussion work happened here:
> [@Discourse should ignore if a character is accented when doing a search](https://meta.discourse.org/t/discourse-should-ignore-if-a-character-is-accented-when-doing-a-search/90198):
>
> Hi. In French, we have a lot of accented characters, and also a lot of people that don’t bother typing them when writing. Because Discourse seems to be strict regarding accents, this leads to partial results only when a user enters a string which can contain accents, from the search feature or the category filter when creating a topic : And And so on… In the topic creation: But: I think this is a major flaw regarding the search efficiency …
Closing this one in favor of ⏫
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [31 באוגוסט, 2018, 1:25am UTC](https://meta.discourse.org/t/diacritics-and-search/73744/18 "2018-08-31T01:25:54Z")