Simple plugin that offers an HTTP POST endpoint to mailgun or any other mail service


(Andrea Bedini) #1

So … I made a simple plugin that offers an HTTP endpoint to mailgun or any other service that is able to POST the raw email (almost, a single parameter name has to match, see below).

I’m not going to tell you how to use it but how to test it.

  • You can use the sandbox domain that mailgun gives you.
  • Setup discourse as described in Discourse as Your First Rails App (yes, it is my first rails app).
  • put mailgun sandbox credentials in config/environments/development.rb, look for config.action_mailer.smtp_settings.
  • you are going to need to tunnel the http endpoint to the greater internet. I used ngrok. Remember that Discourse runs on port 3000 inside the vagrant box but it is mapped to port 4000 on the host.
  • add a route on mailgun that looks like this
match_recipient("(.*)@sandboxblablabla.mailgun.org")
forward("http://blablabla.ngrok.com/plugins/incoming_email/mime")

where blablabla.ngrok.com is the url provided by ngrok. ngrok also provides a real-time web UI where you can introspect the traffic, it should be available at http://localhost:4040.

  • now config your discourse instance. Set reply by email address to something like replies+%{reply_key}@sandboxblablabla.mailgun.org. Enable reply by email enabled and email in. Options log mail processing failures and email time window mins might also be useful.

  • create a couple of users and start posting and replying to messages. Email in also works, just use email addresses with domain @sandboxblablabla.mailgun.org.

Now, beyond the proof of concept:

  • the only specificities to mailgun are 1) the endpoint has to end in mime to signal that the raw mime message is desired. 2) the request is still application/x-www-form-urlencoded with the message in body-mime parameter.

  • is there any standard for the plugins routes? I used /plugins/incoming_email.

  • the plugin accepts any email (and perhaps reply with a failure message) just like an external pop3 service would do.

  • security, would it make sense to add a secret key to the endpoint url (like plugins/incoming_email/{secret_key}/mime/ to prevent abuse?

If anyone is interested in this I can put it on a github repo.

Mailgun documentation for reference.


Set up Reply via Email Support :e-mail:
(Robert) #2

I would like to be able to install your plugin via the docker yml config. :slight_smile: I think having a github repos is a prerequiste.

My 1€/month mail provider allows me to pipe mails to apps and I could use curl to push this to your API. So this is indeed an interesting plugin.


Whereis routes.rb?
(Sam Saffron) #3

We already have an http endpoint for mail processing, a plugin should not be required


Whereis routes.rb?
(Andrea Bedini) #4

What ?! why the hell did I write my own plugin then :blush:

see Mailgun plugin for Discourse and let me know how it goes! You might as well fork it and adapt it to your mail provider.


(Robert) #5

:open_mouth: I checked the API thread and the routes file, but I couldn’t find any thing about it. I guess @andreabedini checked it as well before he started his plugin.

Would you please point us to an example / API documentation?


(Sam Saffron) #6

But I don’t have an example at hand


(Robert) #7

Thanks so much! I wasn’t sure about this one. I don’t know the resources/collection rails API. Maybe @andreabedini can come up with an example. I guess he is much more proficient knowing that he wrote already a plug-in. :wink:


(Andrea Bedini) #8

Thanks @sam, @rriemann I had a quick look and the only thing you need to figure out is how the email message is passed in the POST request. It is passed in a parameter :email you might be good to go. I am on holiday now so I won’t have time to check this out for a little while.


(Robert) #9

Hey @sam,

I tried to get it working with this curl / perl command:

cat mail.mbox | { echo -n "email="; perl -MURI::Escape -lne 'print uri_escape($_)'; } | curl -ivvX POST -d @- "http://forum.my-domain.local/admin/email/handle_mail?api_key=_my_key_&api_username=system"

What’s happing

  • Later, I will need to pipe in the mail. Here, I use cat mail.mbox for testing.
  • I need to assign the raw mail to the email POST param. So I have an echo in there.
  • The perl command reads from stdin coming from the cat and does the urlencoding. Feel free to find a better command (maybe in ruby). I just copy’n’pasted this from Google.
  • everything is piped to curl that will add the api_key and username params to the request and sends everything to the /admin/email/handle_mail endoint.

The Result: An error

NoMethodError (undefined method address_list' for nil:NilClass) /var/www/discourse/lib/email/receiver.rb:50:inprocess’

So something is wrong here. :disappointed_relieved:


(Régis Hanol) #10

What does the email you’re passing in look like?


(Robert) #11
Content-Type: text/plain; format=flowed
To: mail-for-discourse-category@test.local
Subject: Application 2016: Zemlak-Skiles
X-Mailer: nodemailer (1.10.0; +http://www.nodemailer.com; Stub/1.0.0)
Content-Transfer-Encoding: 7bit
Date: Sat, 19 Dec 2015 16:59:22 +0000
Message-Id: <1450544362394-b7768406-590d4c94-e104eaf9@localhost>
MIME-Version: 1.0

Pseudo: Zemlak-Skiles
Roles: Minister, Member of the Parliament (MEP)

Form Data:
{
   "birthdate": "1990-06-05T22:00:00.000Z",
   "nationality": "France",
   "residency": "France",
   "gender": "male",
   "idtype": "Passport",
   "idnumber": "iue",
   "institute": "HU",
   "degree": "masters graduate",
   "studyfield": "eu",
   "studyyear": "2 year(s)",
   "englishreading": "Advanced",
   "englishwriting": "Advanced",
   "englishspeaking": "Advanced",
   "mothertongue": "German",
   "otherlanguages": "French",
   "confirmTerms": true,
   "remark": "",
   "pseudo": "Zemlak-Skiles",
   "submitted": "2015-12-19T16:59:22.386Z",
   "motivation0WordCount": 3,
   "motivation1WordCount": 3,
   "essayWordCount": 492
}

Files:
http://demo.local/files/applications/.Zemlak-Skiles/bestaetigung.pdf

Motivation 1 (words: 3):
here i go


Motivation 2 (words: 3):
Here you go


Essay Question: Is it feasible and desirable for the European Union to rely 
exclusively on the use of renewable energy?
Essay (words: 492):
Lorem Ipsum is simply dummy text of the printing and typesetting industry. 
Lorem Ipsum has been the industry's standard dummy text ever since the 
1500s, when an unknown printer took a galley of type and scrambled it to 
make a type specimen book. It has survived not only five centuries, but 
also the leap into electronic typesetting, remaining essentially unchanged. 
structures, to generate Lorem Ipsum which looks reasonable. The generated 
Lorem Ipsum is therefore always free from repetition, injected humour, or 
non-characteristic words etc.

(Régis Hanol) #12

Why is there no from header?


(Robert) #13

Thank you! I used an early version of the mail.mbox. It’s an obvious bug. :facepalm: The mails is now arriving.

There is just one thing left concerning the formatting. It’s kind of arbitrary. I have plain-text mails that come with some json code. I put it into back-tick fences and it will be shown as code, but doesn’t preserve the white spaces. Citations using > don’t work either.


(Robert) #14

The issue is the newline encoding. The urlencode of curl seems to cut the data after the first newline. Perl doesn’t seem to encode the newline. It workes with broken new lines if I use dos encoding and not working at all for unix encoding.


(Robert) #15

Solution

Eventually I got it working.

My hoster http://uberspace.de/ (chose your own prize) has a qmail based setup that allows to define a catch-all program that gets piped in arriving mails.

So I have a file .qmail-forum-default with this content:

# https://wiki.uberspace.de/mail:dotqmail
| curl -X POST --data-urlencode email@- "http:/your.domain.local/admin/email/handle_mail?api_key=apikey&api_username=system"

The important hint came from Stackexchange.

If the sender mail address is not a known Discourse user, the message will be wrapped in a quote posted by the system user. This destroys mostly the formatting. To get this working, I impersonated the system user and changed its mail address that I’m using now instead to send notifications to Discourse via mail.

I propose that @sam adds this to the documentation thread for the Discourse API.


(Régis Hanol) #16

Or you could enable staged accounts to automatically create new users based on their email address :wink:


(Robert) #17

staged accounts are not working for me. I get a 403 error. Apparently it is incompatible to another setting of my installation. I don’t know which one. Maybe activation or so.


(Robert) #18

At the moment, this is broken because the validation prevents enabling “reply by mail” if pop3 polling is not activated.

I reverted temporarily this commit:


(Robert) #19

@zogstrip Do you plan to change the validation method? Maybe the pop3 polling checkbox could be replaced by a dropdown menue to chose between pop3, no mail support and manual api push?

Otherwise, I would need to finde a solution to automatically revert the commit during Discourse bootstrap. I guess I can just add a line in the section where plugins are usually pulled?


(Sam Saffron) #20

Agree we should have a way of disabling the validation