Rejected mails are unreadable

AI-generated summary

Users discussed a bug where rejected email bodies in the /admin/email-logs/rejected section appeared as unreadable Base64 gibberish instead of the original UTF-8 text. lilydjwg reported that while the database stored the data in a damaged state, the original emails from providers like Tencent and Netease were intact. lilydjwg identified the root cause: the Mail library’s part object had correct charset information, but the body was incorrectly treated as ASCII-8BIT by the email cleaner module. supermathie noted that a fix had recently been merged in PR #37125 and requested a sample email to verify the issue. lilydjwg provided a sanitized test case, but upon supermathie’s inquiry, lilydjwg confirmed that the problem was already resolved in the updated version.

When I check my self-hosted site’s /admin/email-logs/rejected, I find that rejected mails have their body like this:

The correct mail body starts with 谢谢 in UTF-8 (the full body is much longer), and the damaged text is m6Mtyx7x7j+rjuVupyY= in base64.

The mails in the database are already in this damaged state. Comparing that one with the one from the mail server, I find that the mail was rewritten, and I suspect that the email cleaner module scrambled the mail. (But I don’t have enough Ruby knowledge to understand its unique string encoding feature.)

Any chance these messages come from an Exchange server..?:

2 Likes

No. They come from Tencent and Netease. The original mail (I kept it on the mail server to investigate this issue) has no issue with either mutt or Ruby’s Mail lib.

I recently got a rejected email from an Exchange server, and on the contrary it didn’t have this issue.

I got it. The part of a Mail has the correct charset information but somehow its body doesn’t get it and treat the bytes as ASCII-8BIT.

sed -i 's/part.body.decoded/part.body.decoded.force_encoding(part.charset || "utf-8")/' /var/www/discourse/lib/email/cleaner.rb
1 Like

I recently fixed this problem with:

https://github.com/discourse/discourse/pull/37125

Is this email (assuming it’s one that did not get mangled by Discourse) something you can pass to me so that we can investigate further and possibly use it as a test case?

1 Like

I’ve created a stripped-down version of the mail that should exhibit the issue (checked at irb) but without anything private:

issuemail.txt (857 Bytes)

1 Like

Does the email you’re talking about (and this example) still exhibit problems after updating beyond the change I linked?

No, it has been fixed. Thank you.

2 Likes