Webhook to forward posts to a mailinglist: best practice?


(Christoph) #1

In an ideal world, everyone you want to communicate with just signs up for your discourse forum. In the real world, you end up with your discourse forum as the main platform for your communication activity plus a number of additional channels such as a mailinglist of people who refuse to make the transition to the forum.

Has anyone addressed this kind of situation by setting up a webhook that forwards new topics/ posts from certain categories to a mailinglist? If so, I’d be interested to hear about experiences with that, including pitfalls to avoid in that kind of setup.


"Daily Updates" Mailing List Mode occasionally not sending
Discourse Narrative Bot Beta Feedback
(Matt Palmer) #2

Best practice would be to identify what stops the hold outs from using Discourse as a mailing list, and then report bugs / submit PRs to get Discourse to be good enough to be a mailing list replacement.


(Sam Saffron) #3

In lots of cases the “problem” is the mailing list users are werewolves and we are unicorns :unicorn: .

The holdouts are quite obvious and not trivial to resolve

“to address” is targeted at a reply key instead of actual email addresses of people involved.

This means you can not “directly” reply to people or see emails.

We ship lots of chrome which mailing list people don’t like

vs

Our titles have a lot of noise


(Matt Palmer) #4

Anyone who doesn’t like formatted e-mails is almost certainly using a plain-text MUA (or a mailer configured to prefer plain-text), so extra cruft there isn’t a huge issue. As for noisy titles, every damned mailing list crufts up their subject lines with all sorts of crap, so everyone I know who’s a mad-keen mailing lister has procmail rules (yeah, we all use procmail) to rewrite subject lines to remove it all. As for the To: addresses, I’ve noticed a number of mailing lists start to futz with that, too (Mozilla, I’m looking at you, but some google groups appear to be playing similar games), so I don’t think we’re Robinson Crusoe there, either.

Perhaps I should start using meta entirely via e-mail, and see what happens… are you ready for the Womble Rageocolypse of bug reports? :japanese_ogre:


(Sam Saffron) #5

Oh yeah! Make the :volcano: of rage overflow.


(Christoph) #6

Gentlemen, what you are saying is probably all correct, especially for hard core resisters, i.e. active resisters who have strong reasons for resisting. But I’d like to add another type of resister to to the picture and that is the inert type, a kind of passive resister whose resistance is not at all specific to discourse but a broader resistance to change, perhaps paired with a degree of technophobia or unwillingness to spend too much time with communication technology at all. These people have come to use email because they had to and are struggling with “too many emails” because they don’t quite know how to unsubscribe from newsletters/mailinglists or perhaps they do but somehow choose to nevertheless delete emails they don’t want manually.

tl;dr: apart from nerd resisters :nerd:, there are also those people who still have a real life and just can’t be bothered :innocent:.

My concern with this topic was not so much to address the former but to somehow drag along the latter until they find it more convenient to simply use discourse (something the active resisters might never do).


(Matt Palmer) #7

So let’s find out what those strong reasons are, and fix them.

As for the passive resisters, why would they care if they just got seamlessly migrated to a new communications medium, if their user experience wasn’t materially impacted?

Maybe after some DC builds are done, and there are sufficient people available to deal with the runoff…


(Tobias Eigen) #8

this describes 90% of my community.


(Christoph) #9

Yes, force-migrating people (i.e. creating discourse accounts for them without asking for their consent) might be one way to go. Technically. And for some communities this may be acceptable. But I wouldn’t say it’s ethically the best way to do it.

Firstly, if people have signed up for a particular mailing list (and given their consent to receive the emails circulated on that list), that should not be taken as consent for signing up to a discourse forum and even less as an agreement to the ToS, Guidelines, and Privacy Policy of that forum (even if your mailing list happened to have all of these (very unusual), chances are that you will want to adapt them to the new forum environment. To me, all this means that the correct way of migrating people to discourse is via invites.

Seconddly, I’d imagine that it is not totally unusual that you want to merge multiple mailing lists into one forum (in the case of basic mailman or sympa lists) or that you have a segmented list à la Mailchimp where you target different audiences with different newsletters or what not. In both cases, your former mailinglist (or mailinglist segment) will probably translate into groups that are watching their respective categories. If you just force move everyone over to discourse, you’re really pushing them into a rather different environment that what they had before, especially with the summary emails that may have little to do with their original topic of interest. - Yes, they can turn those off with a few clicks, but then we’re back to the “couldn’t be bothered” part.

So, as much as we’d like to believe so, the migration to discourse is definitely not “seamless” and I don’t think it can be seamless in principle (because discourse is different, that’s what makes it great!) and that’s why I’m trying to find ways of transitioning across the seam as proposed in the OP.

I don’t mind this interesting discussion and I’m happy to hear that I’m not the only one struggling with this kind of userbase:[quote=“tobiaseigen, post:8, topic:63609”]
this describes 90% of my community.
[/quote]

but, just as a reminder: the intention with this topic was to identify some best practices for a gradual transition “across the seam”. In particular, I am curious about, well, see title/OP :wink:


(Matt Palmer) #10

If you’re not moving people to the new platform, you’re not migrating, you’re creating another community, and hoping that people come across. Pretty much, that doesn’t work. Never has, never will, and webhooks won’t save you.

The idea that you “can’t” move users between platforms is utter bollocks. Migrations happen all the time between platforms, and as long as the platforms are fairly feature-compatible, rarely does it cause more than a bit of temporary butt-hurt amongst a few people whose cheese has been moved. Now, if you completely change the way the system works (e-mail to web-based forum), yes, that will rightfully piss people off, which is why I’m wondering what features your userbase actually cares about, so we can fix it.

ToS, etc are red herrings; you don’t have to have the default ones if you want, and you can have none at all if you really want to go wild. So is “segmented audiences”; Discourse isn’t mail-a-kimp, and I hope like hell it stays that way. Discourse has categories, which map nicely to individual mailing lists, and that’s that. If you migrate a mailing list properly, people aren’t going to be getting “summary” e-mails, because they’re getting mailing list mode instead, so it makes no sense that you bring that up, either.

You’re not trying to transition “across the seam”, you’re trying to make life hard for yourself by building a second, separate community and then trying to lure people across who don’t like the taste of your bait. I’m yet to see an example of that working, and I’ve seen plenty of examples of it failing miserably. You’re not going to find easy “best practices” to do something that is, itself, a really, really bad idea.


"Daily Updates" Mailing List Mode occasionally not sending
(Christoph) #11

If you have a group of people who are not interested in the entire forum but only their category, mailing list mode is even worse. But I know/hope that [mailing list mode per category]
(Apply "mailing list mode" per category) might eventually be supported, so this may become a non-issue in the future.

I’m also interested in negative examples. So those cases you’ve seen fail, they tried to build a bridge by forwarding certain emails to an outside mailinglist?


(Jeff Atwood) #12

How so? Just mute all other categories. Only a problem if new categories are added later, which also must be muted.


(Happy Lee) #13

Hi @tophee - By “webhook” I have used a distribution group email address for a Discourse account. Only problem I’ve encountered is this: The distribution group must contain all valid email addresses. For domain email, we use Google Apps for Education here - and if too many bounces occur when sending to that distribution group, Google spam controls are quick to permanently blacklist the sending address (configured on the Discourse server). To fix this, I’ve had to modify the sending address of the Discourse server.

I wouldn’t say this is a best practice - it’s more of a hack. However, it’s worked for us the last year (and I hope for years to come).


(Christoph) #14

So you create a new user on discourse, make sure it is watching the categories you want to forward to the mailing list and then you change the users email address to your mailing list address (e.g. mymailinglist@listserv.domain.com)?

I’m not sure I understand what you mean here? What do you need Gmail/Google Apps for?


(Happy Lee) #15

Instead of Microsoft Exchange, postfix, exim, or qmail we use Gmail in Google Apps for Education as our MTA. Our Discourse server is designed to only serve addresses in our domain. Email from our on-campus Discourse server goes out our SMTP gateway to the internet to Google’s Gmail. MX records for our domain point at Google’s mail servers.


(Christoph) #16

Are you saying that you are using Gmail as an outgoing mailserver for your entire discourse? I don’t think that’s a good idea. You should use sparkmail, mailgun or the like for that. Or setup “Straightforward direct-delivery incoming mail”.


(Happy Lee) #17

Not exactly. Gmail is not our outgoing mailserver - Discourse emails go out the Discourse server through a SMTP gateway (on campus here). Gmail is the place we receive inbound mail (MX records point at Google’s mail servers).


(Christoph) #18

Ah, so if discourse sends an email to the distribution list user and one of the emails on the distribution list does not exist, the bounce email will go to the notification email in the discourse site settings (probably something like noreply@mydomain.com) and that email address is routed to the gmail account, right?

Now I’m wondering whether I have my instance configured wrongly because I actually don’t have the notification email address routed to anywhere. In other words, I am not receiving any bounce messages at all. It looks like that solves the problem you’re describing, but maybe it creates other problems?


(Matt Palmer) #19

Yeah, not receiving and processing bounces will end up giving your sending IP a bad reputation, because continuing to try to send e-mail to non-existent addresses is a sign of a spammer, so gmail, hotmail, et al will mark your IP as being a bad actor, which means more of your e-mails go directly to spam, do not write Go, do not collect 200 UIPs.


(Christoph) #20

I have followed the email setup guide written by Jeff and I have setup bounces@mydomain.com according to the instructions provided by sparkmail. Does that mean I am not receiving any bounces (I checked the actual email account and there are no error emails, although that may well be because no errors. occurred)