The discussion revolves around an issue with Vietnamese text not displaying diacritics in search results on a Discourse forum. Pham_Quyet_Nghi initially reports the problem, providing an example of how the text is displayed without diacritics.
sam acknowledges the issue, explaining that it’s a significant problem for Vietnamese communities due to the language’s tone-based nature. They suggest making diacritic stripping optional and disabling it for Vietnamese communities. sam also mentions that excerpts should be generated based on cooked text rather than normalized text.
Pham_Quyet_Nghi shares an example of a page displaying bookmarks in search results, asking where the configuration for this can be found.
sam informs that the option will be added soon and explains why old search results may appear correct. sam later announces that the issue has been fixed with a commit to the Discourse repository and provides instructions on how to rebuild and reindex the search results.
Yes this is pretty tricky, I can see how it is a problem for Vietnamese communities, the excerpts must look very confusing.
In Vietnamese is there ever a reason to type without diacritics? eg type khong and mean không? I imagine it is a super duper hard no cause the language is tone based, so the is the equivalent of me typing “dog” and meaning “milk”.
I think the best way forward here is to make diacritic stripping optional and turn this off in Vietnamese communities.
A bigger and more complex change is to amend it so excerpts are generated based off cooked and not the “normalized” cooked text.
FYI English speakers:
dấu: A sign
dàu: Head
dãu: Pudding
So yeah this is a pretty giant issue for Vietnamese.
The reason it looks “good” on old search results is cause they have not been indexed yet using the new algorithm. If you edit any of the posts there the bug will pop up.