Headers for email notifications so that Gmail users can filter on tags?

Feature
,
1

The post got closed as a duplicate Email: mail headers should have tags in them, possibly in reference to Customs email headers and/or subjects tags, but the latter is kind of general, and I’d like to talk about a specific problem we have:

We’re using tags as the most-logical replacement for dozens of mailing lists about various project sub-topics. We started using categories, but that became unwieldy — categories are heavyweight, don’t allow easy cross-posting, and for a whole other number of reasons, we think tags are a better match.[1]

However… there’s a problem. While it is possible to put %{optional_tags} in the template for email notifications, Gmail’s very limited filtering doesn’t do anything smart with this — and, even for the old-school mail user it’s kind of a pain to write a procmail rule that breaks down the subject line and parses it.

So, it’d be nice to have the tag somewhere else. For the procmail folks, a custom X-Tags header would do, but Gmail won’t care, so we need something else.

One idea would be to actually use tags as List-ID, but I’m not sure Gmail does the right thing with multiple List-ID tags mutliple List-ID fields aren’t permitted [2]. Another idea, which is a bit kludgy but I think would work: put tag@subdomain.example.org (and potentially also tag2@subdomain.example.org, tag3@subdomain.example.org, …) in the CC list. Google can filter on this, and most other systems also have reasonably-advanced features for dealing with CC as a multi-value field. And, we’d redirect any mail actually sent to these addresses to the void[3].

As a bonus, the CC approach could be used as a way to add tags on incoming mail (see Add tags by email). Again, kind of kludgy, but then, so is all email, in 2022.

  1. If you’d like to discuss this, I can… in some other thread topic! ↩︎

  2. curse you, RFCE 2919! ↩︎

  3. i.e. /dev/null ↩︎

2 Likes
2

I’m open to any other suggestions or ideas anyone has. We can’t reasonably suggest a large-scale move to Discourse without this.

3

Also, why isn’t there #tags, here? :slight_smile:

4

It sounds like they might work, but you’ll need a custom plugin.

5

What is your use case, exactly?

I’d be interested to learn why you feel the need to help people filter email notifications from your site. The intended usage of discourse is that community members just use email notifications to be informed when there is activity they care about and click the link to log in when they want to particicipate in discussions. For sites that want it, replying by email can be enabled. But the member should still be in the habit of logging in to continue the discussion on the site.

That way everyone benefits from and can contribute to the collaborative effort of organizing and managing discussions on the site. And you don’t end up with a bunch of out of dated discussions siloed in everybody’s email folders. When discourse is used as intended, the notifications received by email can just be deleted.

Email is also notoriously hard to get right, and not just for discourse. The more you can get your community logging in and engaging on your site, the more successful (and easy to manage) your community will be.

Maybe discourse isn’t the right choice for your community, or you need to keep a few mailing lists operational while you build up your discourse site? I know it can be hard to change habits of longstanding communities.

Good luck! :sunflower:

2 Likes
6

I think that might be right.

You still have procmail folks? And Gmail folks, and Discourse folks? Keeping all three of those groups happy will be very difficult indeed.

If you’ve got a budget of $2000-5000, then you might get a custom plugin that does what you ask, but it’s hard to imagine that it’ll be satisfactory for anyone. You don’t know what other issues will arise once you resolve the ones that you know about now. I’m reminded of managing mailing lists when a whole bunch of people used lan-based email gateways that didn’t follow RFC-822 (that’s back when I used procmail and emacs rmail). The problem is simply untenable.

I would recommend just going with categories. Or, if you really want mailing lists that follow conventions of a couple decades ago, just stick with what works.

2 Likes
7

If these features already exist in other platforms which are rooted in the traditional mailing list model you will probably have more success there, instead of asking they be retrofitted onto a product which is more future-facing.

Or, to put it another way, if this is a dealbreaker why consider Discourse to begin with? Could HyperKitty be extended to add the functionality you need?

1 Like
8

Goal

Currently, Fedora has hundreds of mailing lists, with about 90 with some degree of activity, and a handful that are very active. I want to consolidate that all in one place, which includes bringing our contributor community along successfully. If there is a better option than Discourse for this, no one has created it yet.

Short version

I’ve been actively working on this for three years and thinking about it for at least ten. In talking to people in my community about what blocks them, this specific thing has come up repeatedly.

Long version:

Around the same time Discourse started[1], we created a Mailman3 GUI front end called Hyperkitty, meant to be a modern web UI that people could use to access the underlying mailing lists. You can see this in action for the Fedora Devel List.

Hyperkitty has some neat ideas, but wasn’t really funded at the scale necessary to succeed, and ended up launching with the initial design and no provision for improving and fixing it in real use. And, it takes email as the underlying baseline, and that really tied things down[2] — even if we’d had the resources, keeping to that as the greatest-common factor[3] would have made a frustrating limit.

So I understand where you’re at with this. If you take Wayback Machine journey over discourse.org’s history you can see[4] that Discourse leaned pretty hard into taking lessons learned from both forums and mailing lists and replacing both…

2013

… and that’s mostly survived through today, although there’s less other talk about mailing lists in the various pages. You’ve gone through the same thing that we would have had we resources to invest in Hyperkitty — the email as too-low-highest-baseline problem — and come to the logical conclusion. I totally understand where you’re coming from in now explicitly saying that getting people to the website is the proper use.

Currently:

  • We have dozens of active mailing lists
    • with hundreds of active participants
    • and many thousands of passive subscribers.
  • These lists go back literally over 20 years.
  • Many old-school open source people are really attached to this way of working.
    • it’s familiar,
    • already set up, and
    • arrives into a daily routine without any need to add “check some website”
  • many people are active across different parts of the project, but that “footprint” is very individual

But:

I. These lists are less functional than many people think they are:

  • moderation is next to impossible (all-or-nothing big stick at best)
    • despite efforts, people don’t always keep to the standards we expect
  • mega-threads are not good discussion
  • off-list harassment is easy to launch and out of our control
  • cross-posting is a mess, as subscriptions aren’t consistent
  • impossible to keep up with unless you’re committed
  • people who should participate don’t for various mixes of the above reasons

II. Email is not the future

  • Mailing lists largely opaque to search engines and do not look like “real activity” to most of the world
  • New people do not want to sign up for mailing lists.[5]
  • Mailing list “culture” is not really a thing anymore.[6]
    • And Gmail’s web interface is actively hostile to traditional conventions like inline replies.

III. Email at large is doomed

  • Big providers have the scale to “solve” spam for themselves, and now have anti-incentive to solve it globally.
  • Small providers have decreasing chance of delivering reliably.
  • Mailing lists inherently re-publish, and all of the signing and verification infrastructure doesn’t really care.
  • Companies are switching to Slack and the like for functional communication, leaving email for announcements and broadcasts.
    • and Jira and github and so on for workflow-focused interactions.
  • Again, “normal” people aren’t using it for anything but getting notices about various from companies they are customers of. It’s not really for personal communication anymore.

But there’s still a need

We have real-time conversation covered[7], but we still need the long form, asynchronous conversations that mailing lists have provided. Chat doesn’t cover everything, doesn’t work well globally and with volunteers with varying time commitments. And workflow tools are too narrow.

Discourse really is the best option

  • Mailing lists aren’t a viable future.

  • Hyperkitty is stuck in 2014.

  • We have too much to just use Github / Gitlab.

  • Other possibilities don’t cut it:

    • Ponymail suffers from the same email-as-GCF problem
    • Vanilla is not great. I’ll just leave that there. :slight_smile:
    • Google Groups is the worst of everything.

  • On the + side for Discourse: many other open source communities are consolidating around it. Notably: Python, GNOME…

Enter Cassandra

Not the database — I mean, telling people of doom but no one believing. I hear a lot of “Email works fine”, and “I don’t see a problem with mailing lists”, and, of course, “I hate forums”, or even specifically “I don’t like Discourse”.

But, we really do need change.

So…

I need to get a large, active, important open source community to move their primary project communications platform to Discourse, and a lot of people are skeptical. It’s a big change. I want to make that as easy as possible, both to make it easier and nicer for the people who are skeptical but willing to try and to make it possible to try for people who have email interaction — including filtering — as a personal blocker.

I think that once they’re there, many people will adjust their behaviors — we’ll get more people discovering that directly interacting with the site isn’t so bad.[8] And we have our own project-wide notification system I’ve got plans to tie in, and hopefully that can eventually give people more of what they really need.

But in the meantime, I need to give people what they’re asking for.

  1. I actually tried to suggest Discourse as an alternate approach at that time, rather than rolling our own. Had I a time machine I might go back and push that even harder. But I was not in my current role at that point and the die had been cast… ↩︎

  2. Similar lesson from LUGNET, I think the most amazing and sensible forum software from the 90s — but tied to NNTP as a backend. ↩︎

  3. I know the idiom is “least common denominator” but that makes no sense. Like “I could care less”, but now also with math. ↩︎

  4. I mean, if you don’t already remember, of course! ↩︎

  5. In Korea, email is for old people has come for us all! ↩︎

  6. I can’t remember the last time I heard someone say “netiquette”. ↩︎

  7. using Matrix, at https://chat.fedoraproject.org/… ↩︎

  8. Although, we definitely have this person, so it won’t be everyone. ↩︎

5 Likes
10

Nice writeup! :ok_hand:

Can you describe the filtering you are talking about in a little more detail that people are relying on in their email? I’ve been around for a while too and have run mailman lists myself and participated (and still do) in many mailing lists, but have never encountered automatic filtering using headers. It seems to me that if someone subscribes to a list and they want to filter it into a folder in their email, they’d set that up for themselves on a list-by-list basis. You can do that with discourse too, right?

I think categories work quite well as a replacement for mailing lists, with mailing list mode enabled by the user or the user watching certain categories so they get emailed for every post. Maybe we need to learn some more as well about why you think organizing discussions into tags is better and worth the effort to get parity with how it works with categories already.

Edit: I am reminded of my old post about this, from back in 2014! :scream:

1 Like
11

Sure. The headers Discourse currently sets look like this (from an actual notification from this thread):

List-Unsubscribe: <https://meta.discourse.org/email/unsubscribe/efed8ca1777379c660afc031464b98eb4e6fa2323a71fa12fa2269eeca5b0905>                
X-Discourse-Post-Id: 1216779                                                                                                   
X-Discourse-Topic-Id: 249982                                                                                                   
X-Auto-Response-Suppress: All                                                                                                  
Auto-Submitted: auto-generated                                                                                                 
Precedence: list                                                                                                               
List-ID: Discourse Meta | feature <feature.meta.discourse.org>                                                                 
List-Archive: https://meta.discourse.org/t/headers-for-email-notifications-so-that-gmail-users-can-filter-on-tags/249982       
Feedback-ID: meta:user_replied:discoursemail

If it weren’t for Gmail, adding something like:

X-Discourse-Tags: some-tag, another-tag

See Customs email headers and/or subjects tags - #6 by mattdm — if the email custom headers setting were passed through template expansion so X-Discourse-Tags: %{optional_tags} worked, this part would work already.

And, for procmail and other old-school email client users, this would be sufficient. Unfortunately, for whatever inscrutable Google reasons, Gmail can’t filter on arbitrary tags, and is (as far as I know) limited to To:, From:, Cc: and … fortunately at least, List-ID. Since that’s the 800-lb gorilla, to accommodate those users, setting List-ID by tag instead of category (or, in combination?) is the best I can think of.

However, RFC 2919 says that only one List-ID is permitted per message. So that leaves::

  1. Pick one tag arbitrarily[1]
  2. Generate something including all tags, like List-ID: firsttag_secondtag.discourse.example.org and let users figure that out. [2]
  3. Generate multiple emails per notification, one for each tag and differing only in this header[3]
  4. Leave List-ID referring to category, and instead use the CC idea… [4]

I don’t really love any of those. So as a first pass, X-Discourse-Tags: would at least cover the non-gmail users. (Which is probably a pretty good overlap with web-resistant users, so I think worth doing to see how far that gets us.)

  1. confusing! ↩︎

  2. not great ↩︎

  3. also not great ↩︎

  4. Pretty kludgy. Just adding Cc: %{optional_tags}[4] wouldn’t do it, because it’d need to be expanded so each tag corresponded to a real (black hole) email address. ↩︎

3 Likes
12

Yes, very much so! Your last paragraph sums things up nicely!

2 Likes
13

Very supportive of adding X-Discourse-Tags and X-Discourse-Category (for parity)

I guess long term for Gmail we could add a user option:

  • Please add all tags this topic belongs to in emailed topic titles.

Eg something along the lines of.:

[Discourse Meta] [feature] [email] [notifications] Header for email notifications

Or perhaps when enabled:

[Discourse Meta] Header for email notifications … [feature] [email] [notifications]

5 Likes
14

Yeah, that would be interesting as a user option. We currently have this sitewide:

%{optional_re}%{topic_title} [Fedora] %{optional_pm}%{optional_tags}

We got strong feedback that putting the tags anywhere but last was annoying. And some feedback that google is not very smart about filtering usefully on partial subject lines.

I’m not sure how much we can “fix” Google, (Or rather, I am pretty sure, and it is “not very much”). So there may be some degree of “oh well” that I am going to have to convince people to accept.

4 Likes
15

There are some minor things we can do to improve the situation.

At the moment we are

  1. Limiting to 3 (arbitrary order)
  2. Not wrapping tags in [ ]

I am on the fence but think the arbitrary 3 limit is not great, we should just strip it.

Additionally tags.map{|t| "[#{t}]"}.join(" ") would be better cause then the filter can be on [tagname] vs tagname which is much more likely to appear in the PM title

@martin thoughts?

5 Likes
16

That makes more sense knowing the history. It sounds like you’ve got a budget to make the stuff happen, and some good ideas to get s good bit closer to start. I do think it’ll be hard, and importing all of the data will be a challenge. It’ll still be hard to make all of those people happy and Sam is on board with at least a few of your ideas, so they’ll go into core and not be broken in the future as would likely be the case for a plugin.

3 Likes
17

I agree with you here, though I think rather than removing the limit entirely we can just lean on the max_tags_per_topic setting? I think that would be safer.

Unfortunately, adding [] around the tags won’t help much with Gmail searching, only visually, since from what I can see (e.g. look at Create Gmail filter that contains a special character - Web Applications Stack Exchange, the documentation linked to is no longer there but it is still valid) Gmail strips special characters when searching in the subject and other places. Try searching subject:("[support]") in Gmail, you will get everything with support in the subject not just the ones with square brackets. This is kind of nonsensical of Google to do, they are a search company after all (well, search and ads) but there isn’t anything we can do about that.

We should also be able to easily add X-Discourse-Tags and X-Discourse-Category at the same time in MessageBuilder since we already have those options at this time, and as you say @mattdm this covers the web-resistant users nicely:

I can take this if you’d like?

5 Likes
21

I just merged this @mattdm :

https://github.com/discourse/discourse/commit/c4ea1586563a5e055082c56ff69d7ebbdaa175ee

It doesn’t help a huge amount with Gmail, but this should at least help with terminal-based email users so they can filter these easier.

6 Likes
22

Note @mattdm we really tried here, but gmail is just so difficult. It strips so much from the text when tokenising your hands are tied pretty hard.

Only workaround I can think of is making tag names super ugly in emails:

“This is a demo subject. [Temail, Tnotifications]” (prefix T or some other alpha sequence that gmail “likes”)

But it is a rather ugly and non intuitive solution.

3 Likes
23

Thanks Sam (and everyone!).

I appreciate all of the effort put into this. In the end, there’s only so much fighting against Google’s inscrutable choices we can do.

Yeah, seriously. The only thing I can think to do more would be to add a per-user option:

Make email notification subject lines contain tag names in a super-ugly format which Gmail can filter on.

:-/

5 Likes