"Broken" quoting of/into email

Feature
1

Hi all,

I wasn’t sure which category to put this in - I’m happy to have it moved.

We have a community of around 400 users at present. We migrated from a purely mailing-list format (an old version of mailman, hacked together with OpenLDAP for user authentication, etc), and presented the result back to our membership as “mail-enabled web forums” rather than a straight-up mailing list replacement.

Many of our users still use the mail interface as their primary interaction with Discourse, and a vocal minority are quite, well, vocal, about some issues they see with how Discourse treats email.

The primary concerns they seem to have is that they believe that Discourse breaks quoting and threading of replies. I believe “mailing list mode” largely addresses the latter - so it’s quoting I am focusing on here.

I believe it is possible to “prompt” Discourse to correctly quote emails (or parts thereof) by introducing double carriage returns (I assume this is the trick) between quoted text and comments made inline (seems to only matter after the quoted text, not after the comment before the next quoted section). However, even in my own experience, this seems a bit hit and miss - when doing it myself, and doing it in what I believe is a consistent fashion, sometimes my quotations and comments are nicely separated, at other times they seem to just run into each other (be that within quotes our outside them)

What’s the cause? I’m not sure. I saw this thread:

Which mentions that support for “>” characters indicating quoting was added to Discourse.

I’ve tested doing that, and it seems to work consistently.

What doesn’t seem to work consistently is other characters that get used to indicate quoting - Gmail in particular in my case, which seems to use a vertical line in front of the quoted text.

So, the question is - is there a “secret sauce” to getting Discourse to recognise the quoted text? Can characters other than > be used to indicate quoting - and how many email clients (be it gmail web client or “thick” clients" do we think it’s reasonable for Discourse to be able to “recognise” quoting behaviour from?

1 Like
2

Why is recognizing quoted text important? In terms of email threading, that is determined entirely by mail headers set when someone presses “reply” on a specific post, or replies to an email (same thing, that email represents one post, replying to the email is mechanically the same as replying to the post).

Can you provide some specific examples?

3

It is important when people reply inline - which our membership is particularly fond of doing, particularly in email. I’ve replied to your post in a similar fashion to what our members want to be able to do reliably (I’ve used the web interface to do so - it works consistently here, they wish to do it by email).

Threading, as I mentioned, I believe is largely dealt with by mailing list mode, which I believe we’ve worked to make the default for all of our users.

I believe that getting quoting “fixed” will help us a lot in terms of member satisfaction (and potentially retention). It will also let those of us running the org to focus on other things. :slight_smile:

1 Like
4

I am still unclear what exactly is broken about it – I am afraid that reply does not help me at all.

5

People want to be able to inline-quote via e-mail, with a similar end result as via the web interface (highlight, click “quote”). It’s really unlikely we’d be able to get the results quite the same (extracting the [quote] metadata would be fiendishly tricky), but we could at least recognise common HTML quoting styles, and transmogrify them into the equivalent of how > gets handled in markdown:

Why is recognizing quoted text important?

6

Hi Jeff,

The issue is that when replying via email, the Discourse system seems to merge in-line comments with the quoted text. If you provide me with an email, I’m sure I can provide you with some examples (what was sent verses what Discourse recognised)

That’s exactly what we’re looking for. This would make for some very happy users within my organisation. :smiley:

7

This is actually what I was asking you for – I think you would need to run a mailing list mode only site to really see this and have many real world examples of it?

8

Provide me with an email address, and I shall forward you some examples.

And yes, the majority of our users interact via email.

9

What got sent to Discourse. Note the last few lines in particular.

> | Yet more of what Karl didn't say
> | Yep, you guessed it, more of what Karl didn't say
> | More of what Karl didn't say
>
Single CR before, single CR after.

What Discourse displayed:

As you can see, the “Single CR before, single CR after.” part is contained within a quote, when it should be stand-alone.

10

Yes, turn on the incoming email prefer html setting.

1 Like
11

Thanks. Checking that now. From what I can see, it means that Discourse obeys the HMTL formatting that the email client sends. I guess it requires a level of trust that your users don’t do dumb things…

12

As far as I understand, the setting is for more innocent:

Discourse can use both incoming text-only and HTML-only mails. Text-only mails are simply used as raw Markdown (which does explain the quoting behavior). HTML-only mails are sent through a HTML-to-Markdown-converter, and the result is once again used as the raw Markdown of the user’s post.

When receiving mails which have both HTML and text (which is quite common), Discourse normally uses the text version and ignores the HTML part. This setting changes this behavior to instead ignore the text part and treat the mail like a HTML-only mail :slight_smile:

2 Likes
13

The guideline there is that users must put a blank line between the thing they are quoting and the response. That seems simple enough.

1 Like
14

H Jeff,

codinghorror Jeff Atwood co-founder
July 6

hudrob:
Note the last few lines in particular.

The guideline there is that users must put a blank line between the thing they are quoting and the response. That seems simple enough.

It would seems so, yes - that would be the double carriage return I mentioned earlier. But it doesn’t seem to work reliably.

Sometimes we see it work. Other times, we see the comment merged into the quote. A few times, we’ve seen an entire message after “Hi guys” vanish entirely.

Let me find those example emails…

15

fefrei Felix Freiberger Regular
July 6
As far as I understand, the setting is for more innocent:

Surely it’s all innocent. :slight_smile:

Discourse can use both incoming text-only and HTML-only mails. Text-only mails are simply used as raw Markdown (which does explain the quoting behavior). HTML-only mails are sent through a HTML-to-Markdown-converter, and the result is once again used as the raw Markdown of the user’s post.

That is useful info, thank you.

When receiving mails which have both HTML and text (which is quite common), Discourse normally uses the text version and ignores the HTML part. This setting changes this behavior to instead ignore the text part and treat the mail like a HTML-only mail :slight_smile:

That would explain my testing.

When I sent something which had some text bolded when the setting was off, Discourse converted my bold to italics.

When I turned the setting on, the bolding was honoured. The issue, unfortunately, is that it also stripped out the quoted text entirely.

Thus, I have to assume not a solution in this case. :frowning:

1 Like
16

Do you have that raw email? In my own testing I never saw this happen, unless the quote wasn’t followed by any reply, but I would consider that a feature, not a bug.

1 Like
17

Here is an example.

Email sent:

Email.png1052×363 44.9 KB

Discourse displayed:

Discourse.png952×401 31.4 KB

As you can see, a blank line was inserted between the quoted text and the reply - yet Discourse picked both up as the same quote.

18

Often it’ll look like a blank line has been inserted between the quoted text and the reply, but in the plaintext email your client produces there isn’t one - this is why the raw email is useful in debugging these problems because it’ll show exactly what Discourse received and processed.

I know for sure Gmail has this behaviour, and the solution (if you don’t turn on incoming email prefer html) is to educate your users hit return twice a second time after a quote. That screenshot doesn’t look like Gmail to me, so I’m not sure if that’ll work.