I’m thinking the best way to implement mail-receiver (since I already use AWS SES) is to use the API via a lambda function to call the api endpoint /admin/email/handle_mail
Out there are tutorials on how to take mail to SES and pass it via S3 to a lambda function. Floating out there is some prebaked nodejs functions to then grab that email and relay to a mail server.
But rather than do that it’s seems better to just call the api directly from a lambda function passing the email. (I don’t maintain my own local mail server and dont’ want to use polling anyway).
So I could use a little help either by more detailed documentation (no documentation pages as AFAIK) on calling that endpoint and what to exactly to send (is that a REST endpoint? is there a websocket alternative? format of what to send?) or some code to share. Do I still need to run this extra container? Direct-delivery incoming email for self-hosted sites (that post is now 6 years old)
@dltj here. I can’t help with Node/JavaScript, but I can offer some pseudo-code in the form of the Python that I’m using to make this happen:
""" AWS Lambda to receive message from SES and post it to Discourse"""
import json
import logging
import boto3
import requests
from base64 import b64decode
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
# pylint: disable=C0301
ENCRYPTED_DISCOURSE_KEY = "{cyper-text}"
DISCOURSE_KEY = boto3.client("kms").decrypt(
CiphertextBlob=b64decode(ENCRYPTED_DISCOURSE_KEY)
)["Plaintext"]
DISCOURSE_SITE = "discuss.folio.org"
EMAIL_S3_BUCKET = "openlibraryfoundation-discourse-mail"
DISCOURSE_URL = f"https://{DISCOURSE_SITE}/admin/email/handle_mail"
DISCOURSE_API_HEADERS = {
"Api-Key": DISCOURSE_KEY,
"Api-Username": "system",
}
def log_lambda_context(context):
"""Log the attributes of the context object received from the Lambda invoker
"""
LOGGER.info(
"Time remaining (ms): %s. Log stream: %s. Log group: %s. Mem. limits(MB): %s",
context.get_remaining_time_in_millis(),
context.log_stream_name,
context.log_group_name,
context.memory_limit_in_mb,
)
def lambda_handler(event, context):
""" Handle SES delivery event """
log_lambda_context(context)
LOGGER.info(json.dumps(event))
processed = False
for record in event["Records"]:
mail = record["ses"]["mail"]
mailMessageId = mail["commonHeaders"]["messageId"]
sesMessageId = mail["messageId"]
mailSender = mail["source"]
LOGGER.info(
"Processing an SES with mID %s (%s) from %s",
mailMessageId,
sesMessageId,
mailSender,
)
deliveryRecipients = mail["destination"]
for recipientEmailAddress in deliveryRecipients:
LOGGER.info("Delivery recipient: %s", recipientEmailAddress)
s3resource = boto3.resource("s3")
bucket = s3resource.Bucket(EMAIL_S3_BUCKET)
obj = bucket.Object(sesMessageId)
LOGGER.info("Getting %s/%s", EMAIL_S3_BUCKET, sesMessageId)
post_data = {"email": obj.get()["Body"].read()}
LOGGER.info("Posting to %s", DISCOURSE_SITE)
r = requests.post(
DISCOURSE_URL, headers=DISCOURSE_API_HEADERS, data=post_data
)
if r.status_code == 200:
LOGGER.info("Mail accepted by Discourse: %s", DISCOURSE_SITE)
obj.delete()
processed = True
else:
LOGGER.error(
"Mail rejected by %s (%s): %s",
DISCOURSE_SITE,
r.status_code,
r.content,
)
return processed
I’m using Serverless to manage the deployment and upkeep of the Lambda. In case you want to go a similar route, you’re welcome to start with this serverless.yml file:
how does one create that (encrypted) API key? Does it need to be encrypted?
Do I need to check “manual polling enabled” Push emails using the API for email replies. in order to get the api endpoint to be active or is it so by default?
Anything to pay extra attention to in terms of AWS permissions or SES rule?
The key doesn’t need to be encrypted. The infrastructure code for our project is in a shared space, so I didn’t want to hard-code the key into the source.
I do have “manual polling enabled” on my site although I don’t remember specifically what that does.
Nothing unusual that I can recall with the AWS permissions beyond sending and read/write to the temporary bucket.
Is there someplace in the webui where one generates the api key. Can’t find any place in the settings. Maybe you just set it to what you want as env var when you build? Anyway no clue how to generate an api key.
Yea I saw that but didn’t think it was right because “granular” had no handle_email permission and was unsure which user or all users to choose.
So I’ve generated a key for user “system” with global permission. Is that what you did?
Well, I was wanting to use my “postman” app to learn/try it out but with no documentation I don’t know what to POST in what format(s) & how to pass the key. If I was more familiar with python I could maybe grok this by looking at your code.
I know this is open source project so it’s understandable there is no real documentation on using the handle_mail api only some help from folks like you…thx!
It’s not undocumented because it’s open source, it’s undocumented because no one (or not many) have asked.
Suggestions that you want to POST the encoded email in an email_encoded thingy. routes.rb shows that it’s a POST and the route, which is https://discourse.example.com/admin/email/handle_mail. I got it from the mail receiver sample template (and I’d recommend that you just use that, as if you had, you’d be finished already) but it’s also in discourse/routes.rb at main · discourse/discourse · GitHub (which isn’t immediately clear just what’s happening unless you understand rals).
@dltj I was able to rewrite your code into nodejs using the aws-sdk and got modules. Thanks so much.
There were serveral road blocks along the way, from AWS (ses,IAM,S3,lambda), messing with serverless, getting the API call correct, to getting the discourse settings proper.
What I am going to do is write a detailed topic on exactly what I did to get this working, enough detail that someone could enable this without having to do any to coding or pulling hairs. The real advantage to this method is that by using ses one does NOT need to set up a email server for the domain. This leaves you free to set up a totally independent mail server for your domain.
If you are interested in how it was done check back I’ll post a link here when it’s ready (a few days).
Adding to this, @dltj we will be removing the email param from the admin/email/handle_mail route soon, you need to send the body of the email as a base64 encoded string in an email_encoded param to this route.
encoding was one of issues I could not resolve
I could not get the encoded email to work
@martin per your post email_encoded is incorret. I’ve tried both, email_encoded throws error, but encoded_email still gives the warning.
body: 'warning: the email parameter is deprecated. all POST requests to this route should be sent with a base64 strict encoded encoded_email parameter instead. email has been received and is queued for processing',
Do I need to convert my incoming mail to plain text (which I am doing now) and back to base64 or is it likely that what AWS Ses stores in in the bucket is already base64? If I pass the email body as SES saved it fails to show up in the received email log. The post command returns no error just the warning. So basically I’m stuck here.
Please delay removing email key until we can get this to work.
This is great, thank you for sharing.
We were using the deprecated pre- base64 version for three years on Lambda and it went dark a month ago.
We can’t use the SMTP :25 as we are hosting behind a cloudflared tunnel and why play with port 25 if amazon does such a good job blocking the noise with SES?
Anyway, I’m at the end of my sanity.
The code works with Postman and python executing on a local computer. Something seems to get ruined when being sent from AWS Lambda with identical code.
Forbidden
UTF-8
Well, solved my problem -
even though the the queries were being passed, the ‘bot fight mode’ does NOT play nice with the Discourse API. Maybe because the base64 encode?
Hope this saves someone 72 hours of their life:
Turn this OFF:
