Better email reply parsing 📧

[techAPJ (Arpit Jalan)]

There are many open bugs on meta related to incorrect email reply parsing and I have been looking into them since past few days.

Almost everytime the fix is to correct/modify something in GitHub’s email_reply_parser library.

The GitHub’s email_reply_parser library is now abandoned. The last commit was over a year ago, and there are many open issues / pull requests.

I propose that we include the email_reply_parser in core Discourse email library so that we can customize it as per our requirements, without depending on GitHub’s library.

I have laid the groundwork for this, and in process fixed these bugs:

Email parsing reply not correctly stripping `----Original message----` marker

Email parsing reply not correctly handling ----Original message---- marker. Here is a modified example (to protect privacy): [image] I have two examples of this now with the same user. I’m willing to send via PM the raw original email replies to a one of the Discourse Team to help resolve this. Cheers, Dean.

https://meta.discourse.org/t/html-email-signature-not-being-stripped-out-of-notification-reply/21351

Email-created reply including reply preamble

Reproduce: Reply using Gmail to a post notification. Get a separate rejection error because your reply is less than the default 20 character minimum. Reply to your first attempt to reply (which still is addressed to Discourse) with a longer response. Expected: Only the response content appears. Actual: Your email address is exposed. [image] Pastebin of original message content available to those who request.

Malformed bullet causes emailed in topic to be incorrectly truncated

I know that a line with two dashes truncates everything below it. -- But it turns out that a line with a single dash immediately followed by text (no space) will also truncate everything below it. See screenshots. [image] [image]

Here is the PR for the same:

FEATURE: better email reply parsing

Looking forward to hear @team’s feedback/suggestions.

[eviltrout (Robin Ward)]

Wow I did not realize that library was only about 280 lines of code!

One downside with importing it this way is you are not importing their tests. If we regress on something they already have a test for, we are not going to know are we?

Maybe it makes more sense to fork it as discourse-email-parser, and add your new functionality and tests to the suite in that project?

[zogstrip (Régis Hanol)]

I agree %. Forking is a better idea.

[techAPJ (Arpit Jalan)]

Okay, I just published a new gem discourse_email_parser

Here is the GitHub repo: https://github.com/discourse/discourse_email_parser

and RubyGems.org page: discourse_email_parser | RubyGems.org | your community gem host

Updated the PR to use discourse_email_parser instead of email_reply_parser gem.

[techAPJ (Arpit Jalan)]

This change is now live, @zogstrip just merged the PR!

[fefrei (Felix Freiberger)]

It looks like the parser is not yet trying to detect the “On <date>, <name> wrote”-line in any non-english language:

https://github.com/discourse/discourse_email_parser/blob/dfc6031cfce718e4d0cadd9a52a72be0016e2c55/lib/discourse_email_parser.rb#L89

Is this correct? How can we contribute translations for this to at least cover the most common languages?

[techAPJ (Arpit Jalan)]

discourse_email_parser is now replaced by email_reply_trimmer

https://github.com/discourse/email_reply_trimmer

and it covers common languages

https://github.com/discourse/email_reply_trimmer/blob/c7f36f34afdf3b40b47ec89f01dc4a1a7b8eb194/lib/email_reply_trimmer/embedded_email_matcher.rb#L13-L26

[fefrei (Felix Freiberger)]

That list might soon include German:

https://github.com/discourse/email_reply_trimmer/pull/1