Mailing list (and NNTP) bridge


(Joel Uckelman) #1

Continuing the discussion from NNTP support?:

First, some background: I’m Joel Uckelman, one of the developers of VASSAL, an open-source virtual tabletop program for boardgames. We used mail2forum to bridge our mailing list to our forum back when phpBB 2 was current. When phpBB went to version 3 and mail2forum didn’t, I wrote a mailing list bridge which we’ve been using more or less successfully at our forum since 2009 or so. I would very much like to see Discourse have mailing list support so that switching to Discourse from phpBB is an option for us.

So, some thoughts, in no particular order:

  • Should mailing list support be integrated or work with existing lists? The list bridge I wrote for phpBB is truly just a bridge. Mailing list software (Mailman, in my case) provides the list. There were some advantages to this approach: It was a lot less work, since I could rely on Mailman to handle mail properly, and any PHP (blech!) I didn’t have to write was welcome. On the other hand, bridging forum posts to an existing list means that the list software doesn’t know anything about forum accounts, etc., and as a consequence, users have to subscribe to the list manually, so you can end up with list subscribers who don’t have accounts on the forum. This hasn’t been much of a problem for us because the number of subscribers to our list is small, but I could see there being issues (confusion, mainly) if there were more subscribers.

  • How granular should subscription be? My list bridge supports one list per subforum. It’s not required that each subforum have a unique list, or any list at all. E.g. our Games in Progress subforum isn’t hooked up to any list, the Test subforum has its own list, while the other eight subfora are bridged to the main list. The main list can be something of a firehose at times, but my reason for having it set up this way is that the list subscribers are mostly developers, team members, and some people who are a constant presence answering user questions—these are the people who want to see every message which comes through. But I could see someone wanting a bit more control over what they get by mail, too.

  • How should bounces be handled? What I’m doing right now is letting Mailman handle bounces.

  • Being able to start topics and reply to posts by mail, as well as suppressing reply notifications for list subscribers (you don’t need to be emailed about replies if you’re already geting posts by mail!) are mandatory features. Being able to unsubscribe from the list by mail is close behind, for me. Nothing else strikes me as terribly urgent.

  • There are various things which can come in from the list side which we have to think about how to handle: attachments, quoting, ugly MIME parts (base64, quoted-printable, HTML), bounces, forgeries, spam. If you want to see what I did with, e.g., quoting, browse around the posts in our forum. It’s not the prettiest thing in the world, but it’s functional.

  • Determining where incoming mail goes in the forum is not that hard, so long as we construct Message-Ids from which we can recover post IDs. We do have to decide what to do with incoming messages which aren’t In-Reply-To any message we already have. Most of the time, these will be starting new topics, but it is possible that the message was sent by a broken mail client…

  • I suspect that a lot of the infrastructure for handing email would also work for NNTP. (I think supporting NNTP would be neat, too, but email is my priority.)


Using discourse as a mailing lists
Can Discourse replace mailing lists?
Replacing Mailing lists: Email-In
Discourse as a front end for existing ASF mailing lists
NNTP support?
Using discourse as a mailing lists
(Jeff Atwood) #2

The current thinking is that reply-via-email support is very important, as it achieves a core mission goal: to displace mailing lists which are uniformly horrible.

I am not so keen on the idea that you can create new topics via email, however. If this was possible it would have to be at higher trust levels.

But

hey, someone (quoted you / @name mentioned you / replied to a topic you are watching), here’s an email notification with the text of that reply

should trivially equate to

click the reply button in your mail client, write a reply, click the send button in your email client

and that should “just work”.

Beyond that, we will see, but that’s essential, absolutely, and I miss it every single day when I have to load a whole browser instance and all our JavaScript and resources, just to fire off one dinky quick little reply.


(Joel Uckelman) #3

[quote=“codinghorror, post:2, topic:3453, full:true”]
The current thinking is that reply-via-email support is very important, as it achieves a core mission goal: to displace mailing lists which are uniformly horrible.[/quote]
Mailing lists are uniformly horrible for you. My reason for wanting to post by email is that fora are uniformly horrible for me in comparison with the email tools I use (nmh and vim). I want to displace my use of fora with email; you, vice versa. I don’t see why we can’t have both. Broadly speaking, I want users to have a choice about the tools they use for discussion. That’s a better goal than to displace mailing lists for everyone, including those who would rather use email.

What specifically bothers you about creating new topics by email? We’ve been doing that successfully at my project’s forum since 2009, without a single problem I can recall.


(Sam Saffron) #4

We would like this integrated into core, this means the user would enter pop or imap creds in some config (ideally from the admin section) and the clockwork job would take care of polling and processing the mail.

I would like us to build on the existing bits here. We already are able to “subscribe” to conversations by watching them. We intend to allow users to “subscribe” to entire categories if they wish. This will become much more relevant when we build “secret” categories only a sub-group on the forum have access to.

We should probably flag something on the account, keep in mind, we know that the email works, this would be an edge case (we force a valid email on signup)


There are a few key ideas of mine, in no particular order:

  • We would like to focus on quick iteration of this feature, would be happy to get check-ins with bits and pieced disabled that we can slowly enable. Getting this right will involve lots of real world testing.

  • This is a critical feature for us, its possible @eviltrout or I would like to get started helping out / building bits in a week or two. We would like to have the basic, reply by email done in the next month.

  • As a rule we would like to make this as easy as possible to deploy and get going, this involves a great logging story in the admin section, easy setup and handling of everything possible from within our current infrastructure with minimal dependencies. (clearly not planning to write pop server or smtp server, those dependencies are clearly fine)


Import from Google Groups to Discourse
(Sam Saffron) #5

Keep in mind, Discourse is a system of rainbows, @codinghorror chooses the defaults, but you will be free to customise it to your own needs.

Allowing unregistered users to create topics via email is full of dragons, for example you cant get the to fill captchas or place js traps, if the email leaks out, trouble.

Allowing low trust users to create topics via email is risky cause it becomes very tricky to stop duplication prior to post, which may hurt non mail-list users.


(Sam Saffron) #6

I really want a plugin for discourse that replaces the textarea with codemirror and vim bindings, it would make me so happy :smile:

I get that websites may not be the most productive place to consume information. Our goal is to increase and improve civilised discourse online, giving a mail frontend is very much on mission.


(Joel Uckelman) #7

[quote=“sam, post:4, topic:3453”]
We would like this integrated into core, this means the user would enter pop or imap creds in some config (ideally from the admin section) and the clockwork job would take care of polling and processing the mail.[/quote]
I don’t follow. As a user, why would the forum need my POP or IMAP information, instead of just my email address? Are you talking about the user who is the person setting up the forum here?

[quote=“sam”]

I would like us to build on the existing bits here. We already are able to “subscribe” to conversations by watching them. We intend to allow users to “subscribe” to entire categories if they wish. This will become much more relevant when we build “secret” categories only a sub-group on the forum have access to.[/quote]
That’s fine with me, so long as there’s a setting for users to automatically subscribe to everything.

[quote=“sam”]

We should probably flag something on the account, keep in mind, we know that the email works, this would be an edge case (we force a valid email on signup)[/quote]
My experience has been that this is less of an edge case than you might think. I’ve seen quite a few email accounts go invalid since I’ve been adminning our forum. I concur on the solution.

[quote=“sam”]
There are a few key ideas of mine, in no particular order:

  • We would like to focus on quick iteration of this feature, would be happy to get check-ins with bits and pieced disabled that we can slowly enable. Getting this right will involve lots of real world testing.[/quote]
    I found that the biggest problem was parsing. Everything else was pretty straightforward. One thing which we might find useful is the battery of tests I wrote for my list bridge. They mainly cover message formatting—i.e., this forum post should produce this email message, this email message should produce this forum post.

[quote=“sam”]

  • This is a critical feature for us, its possible @eviltrout or I would like to get started helping out / building bits in a week or two. We would like to have the basic, reply by email done in the next month.[/quote]
    Sounds good. How should we start? I’m not familiar with the D codebase. Where should I be looking? This is all in Ruby, yes? (I’ve never used Ruby before, but, hey, it’s just another imperative language.)

I think the simplest possible thing to do for incoming mail is to have an email address for which everything that’s sent to it is piped to a script for processing. This means that the only thing which has to be set up is the address alias (one line in Postfix, e.g.). Clearly D already has something which knows how to handle outgoing mail, so whatever that is should be reused.

You clearly have something in mind for POP, but I don’t see what. Elaborate a bit?


(Joel Uckelman) #8

[quote=“sam, post:5, topic:3453”]
Keep in mind, Discourse is a system of rainbows, @codinghorror chooses the defaults, but you will be free to customise it to your own needs.

Allowing unregistered users to create topics via email is full of dragons, for example you cant get the to fill captchas or place js traps, if the email leaks out, trouble.[/quote]
No, no! I never said anything about unregistered users being able to create topics by email. (I don’t want unregistered users to be able to do anything but browse.)


(Sam Saffron) #9

Talking about the forum admin here, he/she needs to configure it with the pop/imap creds.


(Sam Saffron) #10

Yes most of that is done with the Ruby mail gem that ships with Rails, luckily it also support POP/IMAP so we don’t need to introduce any new dependencies.


(Joel Uckelman) #11

I use It’s All Text in Firefox, which is not using vim bindings, but using actual vim. But, yeah, I would love to have vim bindings, not just here, but everywhere.


(Joel Uckelman) #12

Ok, I thought you might be talking about the forum admin. But what would this be for?


(Sam Saffron) #13

The way I see it, the forum admin enters the email/password of the “mail” handling account. Discourse already has its own schedular (clockwork) so it could check this account look at messages there process them and integrate into the forum.

We are looking at minimising friction for new installs. Ideally we would even work with a gmail account, that really reduces a big pile of friction. We can use user+routing_id@bla.com to make routing easier.


(Joel Uckelman) #15

Ok, I get what you’re talking about now. I decided against this approach for the list bridge I wrote, for the following reasons:

  • Polling the POP account is either going to be wasteful (when there are no messages) or cause additional lag between posting and post appearing in the forum (when polling is too infrequent). Piping incoming mail to a script means no polling.

  • If you’re hosting all of your own stuff, then using POP has more friction than piping incoming mail to a script. In order for me to to that, e.g., I’d have to set up a POP server, rather than simply adding a one-line email alias. I can see how being able to use POP is an advantage for people not hosing everything themselves, but it’s wrong to say that it’s simpler in all cases.

The difference between pushing mail and pulling mail here is likely only to be a few lines of code—the push code just has to read from stdin and hand that off to whatever we write to parse the mail, while the pull code reads from POP or IMAP and hands off what it reads to the mail parsing function. So, I think it’s feasible and desirable to do both.


(Kevin P. Fleming) #16

Mailing list replacement is a huge requirement for our potential usage as well; it was the reason we were seriously considering Vanilla Forums, but we’ve now dropped that line of thinking for a number of reasons. My list of goals pretty much mirrors @uckelman’s; I’d like for users who want to interact solely via email to be able to do so, transparently to the users who interact via the web UI. That includes category subscription/unsubscription, starting new topics, etc.


(Eli the Bearded) #17

Spammers forging as member users is the problem that comes to my mind. Bad autoresponders (“Hi, you just sent me email, but I’m on VACATION!”) could be an issue.

I have very few mailing list subscriptions left, but some of the ones I do are low-volume, low-frequency ones that are not well served by forums. If there is one message in six months, I don’t want to just have a forum ping me to log in and read that message. I simply want to see it.


(Sam Saffron) #18

cool, so whats the plan on moving forward here, what is you timeline looking like?


(Joel Uckelman) #19

I see two problems here: One is spam from registered users, the other is forgery (which might also be spam). Spam from registered users is the same problem regardless of the method of ingress—whatever Discourse does about it for posts originating on the web should be what it does about it for posts originating by email. Forgery, on the other hand, is only possible for messages coming in by email (assuming, of course, that the authentication stuff on the site is done properly).

So, forgery.

First, I want to describe the setup I have for my existing list bridge. We haven’t had a single piece of spam or forged mail come in over the list bridged to my forum—ever. We’re not invisible to spammers, which I know because we do get occasional forum spam. We might just be lucky—the population who uses the list is tiny in comparison with the population using the forum, and you have to hunt around a bit to even know there is a list. Some bogus mail can be (should be!) taken care of by the mail server, before it gets to Discourse. E.g., my mail server is doing a bunch of checks to reject bogus mail during SMTP transactions—rejecting invalid HELO and sender hostnames, using the spamhaus.org RBL, and greylisting. I haven’t checked the stats recently, but my impression is that the first two things stops mail sent by the total bozos, while the greylisting stops spammers who can’t be bothered to retry sending—which is the the vast majority of them. I could probably cut out even more by checking SPF records or DKIM.

If a forgery gets through to Discourse, at least in the kind of setup I’d have, that means it has headers which more or less check out, and it came from a mailserver that’s not blacklisted and was willing to retry sending the message. One thing Discourse should obviously do is check that the Sender or From match an email address of a registered user. If not, it’s junk—don’t create a post from it. (Should we bounce it back? Maybe that would just cause backscatter… Hmm.) Already at this point, it looks to me like registering a throw-away account from which to spam is less effort than getting everything right to send Discourse spam by email. It might not remain that way if Discourse becomes popular (as I believe it will), though. This also won’t stop a determined forger who is a troll, not a spammer.

Ultimately, the forgery problem amounts to “how do I know who sent this message?”, which isn’t really solvable without public-key crypto. I think the key will be finding a solution that mostly deters forgery without introducing too much friction for users. What does existing list software do about this, if anything? It’s something we should look into. So far as I know, I’ve never seen forged messages which are spam on any list to which I’ve been subscribed.

I’m trying to remember what list software I’ve used does about this. I’ll bet we could look at, e.g., Mailman, for some well-tested solutions.


(Daniel Watkins) #20

They use WITHERING CONTEMPT.


(Joel Uckelman) #21

Well, I need to learn enough Ruby not to be a doofus, but that doesn’t depend on anyone but me. It would help me to have an idea of the architecture of Discourse. I don’t need to know everything, but I need at least some local knowledge of the post-submission API, as that’s what the incoming mail handler will need to feed. Pointing me to this stuff would be welcome.

As for timeline: I have a major project I’m trying to wrap up right now, which I think/hope will not extend into March. The way I tend to work is bursty, so it’s quite possible that once I’m free from this other project, I could work pretty intensely on a list bridge for some days.