Better email reply parsing 📧

techAPJ · 9 בדצמבר,‏ 2015,‏ 6:49pm

There are many open bugs on meta related to incorrect email reply parsing and I have been looking into them since past few days.

Almost everytime the fix is to correct/modify something in GitHub’s email_reply_parser library.

The GitHub’s email_reply_parser library is now abandoned. The last commit was over a year ago, and there are many open issues / pull requests.

I propose that we include the email_reply_parser in core Discourse email library so that we can customize it as per our requirements, without depending on GitHub’s library.

I have laid the groundwork for this, and in process fixed these bugs:

https://meta.discourse.org/t/html-email-signature-not-being-stripped-out-of-notification-reply/21351

Here is the PR for the same:

Looking forward to hear @team’s feedback/suggestions.

eviltrout · 9 בדצמבר,‏ 2015,‏ 8:41pm

Wow I did not realize that library was only about 280 lines of code!

One downside with importing it this way is you are not importing their tests. If we regress on something they already have a test for, we are not going to know are we?

Maybe it makes more sense to fork it as discourse-email-parser, and add your new functionality and tests to the suite in that project?

zogstrip · 9 בדצמבר,‏ 2015,‏ 9:08pm

I agree %. Forking is a better idea.

techAPJ · 10 בדצמבר,‏ 2015,‏ 7:45pm

Okay, I just published a new gem discourse_email_parser

Here is the GitHub repo: GitHub - discourse/discourse_email_parser: Small library to parse plain text email content

and RubyGems.org page: discourse_email_parser | RubyGems.org | your community gem host

Updated the PR to use discourse_email_parser instead of email_reply_parser gem.

techAPJ · 11 בדצמבר,‏ 2015,‏ 2:31pm

This change is now live, @zogstrip just merged the PR!

fefrei · 4 בפברואר,‏ 2016,‏ 9:41am

It looks like the parser is not yet trying to detect the “On <date>, <name> wrote”-line in any non-english language:

github.com/discourse/discourse_email_parser

lib/discourse_email_parser.rb

dfc6031cf


      
          # Returns this same Email instance.
          def read(text)
            # in 1.9 we want to operate on the raw bytes
            text = text.dup.force_encoding('binary') if text.respond_to?(:force_encoding)
          
            # Normalize line endings.
            text.gsub!("\r\n", "\n")
          
            # Check for multi-line reply headers. Some clients break up
            # the "On DATE, NAME <EMAIL> wrote:" line into multiple lines.
            if text =~ /^(?!On.*On\s.+?wrote:)(On\s(.+?)wrote:)$/nm
              # Remove all new lines from the reply header.
              text.gsub! $1, $1.gsub("\n", " ")
            end
          
            # Check for "---- Original Message ----"
            # and strip email content after that part
            if text =~ /^([\s_-]+Original (?i)message?[\s_-]+$.*)/nm
              text.gsub!($1, "")
            end

Is this correct? How can we contribute translations for this to at least cover the most common languages?

techAPJ · 4 בפברואר,‏ 2016,‏ 9:52am

discourse_email_parser is now replaced by email_reply_trimmer

and it covers common languages

github.com/discourse/email_reply_trimmer

lib/email_reply_trimmer/embedded_email_matcher.rb

c7f36f34a


      
          ON_DATE_SOMEONE_WROTE_MARKERS = [
            # Dutch
            ["Op","het volgende geschreven"],
            # English
            ["On", "wrote"],
            # French
            ["Le", "a écrit "],
            # Polish
            ["Dnia", "napisał\\(a\\)"],
            # Portuguese
            ["Em", "escreveu"],
            # Spanish
            ["El", "escribió"],
          ]

fefrei · 4 בפברואר,‏ 2016,‏ 9:58am

That list might soon include German:

techAPJ · 5 בפברואר,‏ 2016,‏ 10:06am

This topic was automatically closed after 24 hours. New replies are no longer allowed.

נושא		תגובות	צפיות
Email reply parsing Bug email	10	3287	30 במרץ,‏ 2016
Email parsing reply not correctly stripping `----Original message----` marker Bug email	10	4058	11 בדצמבר,‏ 2015
Email-created reply including reply preamble Bug email	5	2010	11 בדצמבר,‏ 2015
Where to discuss issues with email_reply_trimmer? Support email	8	840	23 באוגוסט,‏ 2023
Proposed plugin to improve reply-by-email accuracy Feature	2	718	26 בדצמבר,‏ 2023

Better email reply parsing 📧

נושאים קשורים