Garbled text in CSVs with special characters downloaded from Data Resource Manager

zogstrip · 9 september 2026 om 08:58

thanks for the report @singi2016cn will be fixed by

github.com/discourse/discourse

FIX: Add a UTF-8 BOM to CSV exports so spreadsheets read them correctly (#43424)

main ← csv-utf8-bom

merged 11:31AM - 09 Sep 26 UTC

ZogStriP

+260 -225

Reported at https://meta.discourse.org/t/csv/411280 — a Data Explorer CSV contai…ning CJK text opens as mojibake in Excel. The bytes were always correct UTF-8. `Content-Disposition: attachment` writes them to disk verbatim, so the `charset=utf-8` header never reaches the spreadsheet and Excel falls back to the system codepage. Confirmed numerically: the exported bytes decode under CP936 to exactly the glyphs in the report. Every CSV a person opens now starts with a BOM, and every CSV we read back tolerates one. The same root cause was reported in 2023 for the admin user list export (https://meta.discourse.org/t/exported-csv-files-have-garbled-characters/277293) and is fixed here too. ### Also in this PR - Two live bugs, independent of what we emit: an Excel-saved badge award list silently dropped its first username, and an Excel-saved emoji manifest failed every row with `missing_name`. - `poll.gjs` re-adds the BOM client-side, because the XHR text decode strips the one the server sent. - `rchardet` is removed — nothing called `Encodings.to_utf8` any more. Drops an LGPL dependency from a GPLv2 project. ### Notes for reviewers - `CSV.open(path, "w:bom|utf-8")` does **not** write a BOM; Ruby honours `bom|` on read only. The explicit `csv.to_io.write` is required. - `CSV.generate(+Encodings::BOM)` needs the unary `+` — the constant is frozen and `CSV.generate` mutates its seed. - This changes the response body of `run.csv`, which is reachable with an API key. Scripts parsing it should decode as `utf-8-sig` (Python) or equivalent. pandas, polars and duckdb need no change. - It does **not** change the field separator. Excel on Windows takes that from the regional list-separator setting, so `;`-locale users still need the import wizard. ### Plugin authors `paviliondev/discourse-news` calls `Encodings.to_utf8` and `try_utf8` in `lib/news/rss.rb` and will need to inline them. It is the only consumer across all-the-plugins, all-the-themes and both custom meta repos.

Topic		Antwoorden	Weergaven
Exported CSV files have garbled characters UX	0	1178	31 augustus 2023
Ignore BOM on CSV when sending bulk invitations Bug	3	1047	9 mei 2017
Data Explorer cannot export to CSV Bug	11	1792	4 oktober 2017
Gibberish when exporting Support	1	660	28 september 2020
Data explorer uses popup for download UX data-explorer	0	239	9 maart 2024

Garbled text in CSVs with special characters downloaded from Data Resource Manager

Gerelateerde topics