AWS SES / AWS Lambda mail receiver endpoint code?

I’m thinking the best way to implement mail-receiver (since I already use AWS SES) is to use the API via a lambda function to call the api endpoint /admin/email/handle_mail

Out there are tutorials on how to take mail to SES and pass it via S3 to a lambda function. Floating out there is some prebaked nodejs functions to then grab that email and relay to a mail server.

But rather than do that it’s seems better to just call the api directly from a lambda function passing the email. (I don’t maintain my own local mail server and dont’ want to use polling anyway).

So I could use a little help either by more detailed documentation (no documentation pages as AFAIK) on calling that endpoint and what to exactly to send (is that a REST endpoint? is there a websocket alternative? format of what to send?) or some code to share. Do I still need to run this extra container?
Direct-delivery incoming email for self-hosted sites (that post is now 6 years old)

Someone apparently did this with Python How to update mail-receiver to the release version - #7 by dltj? but there is no details and I’ll need to do this in node/javascript.

Does anyone has a working javascript function they can share? Thx

2 Likes

@dltj here. I can’t help with Node/JavaScript, but I can offer some pseudo-code in the form of the Python that I’m using to make this happen:

""" AWS Lambda to receive message from SES and post it to Discourse"""
import json
import logging
import boto3
import requests
from base64 import b64decode

LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)

# pylint: disable=C0301
ENCRYPTED_DISCOURSE_KEY = "{cyper-text}"
DISCOURSE_KEY = boto3.client("kms").decrypt(
    CiphertextBlob=b64decode(ENCRYPTED_DISCOURSE_KEY)
)["Plaintext"]
DISCOURSE_SITE = "discuss.folio.org"
EMAIL_S3_BUCKET = "openlibraryfoundation-discourse-mail"

DISCOURSE_URL = f"https://{DISCOURSE_SITE}/admin/email/handle_mail"
DISCOURSE_API_HEADERS = {
    "Api-Key": DISCOURSE_KEY,
    "Api-Username": "system",
}


def log_lambda_context(context):
    """Log the attributes of the context object received from the Lambda invoker
    """
    LOGGER.info(
        "Time remaining (ms): %s.  Log stream: %s.  Log group: %s.  Mem. limits(MB): %s",
        context.get_remaining_time_in_millis(),
        context.log_stream_name,
        context.log_group_name,
        context.memory_limit_in_mb,
    )


def lambda_handler(event, context):
    """ Handle SES delivery event """
    log_lambda_context(context)
    LOGGER.info(json.dumps(event))

    processed = False

    for record in event["Records"]:
        mail = record["ses"]["mail"]
        mailMessageId = mail["commonHeaders"]["messageId"]
        sesMessageId = mail["messageId"]
        mailSender = mail["source"]

        LOGGER.info(
            "Processing an SES with mID %s (%s) from %s",
            mailMessageId,
            sesMessageId,
            mailSender,
        )

        deliveryRecipients = mail["destination"]
        for recipientEmailAddress in deliveryRecipients:
            LOGGER.info("Delivery recipient: %s", recipientEmailAddress)

            s3resource = boto3.resource("s3")
            bucket = s3resource.Bucket(EMAIL_S3_BUCKET)
            obj = bucket.Object(sesMessageId)
            LOGGER.info("Getting %s/%s", EMAIL_S3_BUCKET, sesMessageId)
            post_data = {"email": obj.get()["Body"].read()}

            LOGGER.info("Posting to %s", DISCOURSE_SITE)
            r = requests.post(
                DISCOURSE_URL, headers=DISCOURSE_API_HEADERS, data=post_data
            )
            if r.status_code == 200:
                LOGGER.info("Mail accepted by Discourse: %s", DISCOURSE_SITE)
                obj.delete()
                processed = True
            else:
                LOGGER.error(
                    "Mail rejected by %s (%s): %s",
                    DISCOURSE_SITE,
                    r.status_code,
                    r.content,
                )

    return processed

I’m using Serverless to manage the deployment and upkeep of the Lambda. In case you want to go a similar route, you’re welcome to start with this serverless.yml file:

service: ses-post-to-discourse

frameworkVersion: "=1.36.1"

provider:
  name: aws
  runtime: python3.7
  onError: arn:aws:sns:us-east-1:{AWS_ACCOUNT}:ses-discourse-DLQ
  region: us-east-1
  iamRoleStatements:
  - Effect: 'Allow'
    Action:
      - 'ses:SendBounce'
    Resource:
      - '*'
  - Effect: 'Allow'
    Action:
      - 's3:*'
    Resource:
      - 'arn:aws:s3:::openlibraryfoundation-discourse-mail/*'
  - Effect: 'Allow'
    Action:
      - 'kms:decrypt'
    Resource:
      - 'arn:aws:kms:us-east-1:{AWS_ACCOUNT}:key/{UUID-of-key}'

package:
  include:
    # - something
  exclude:
    - node_modules/**
    - .venv/**
    - __pycache__
    - secrets.yml

functions:
  emailDispatcher:
    handler: post_ses_delivery_lambda.lambda_handler
    events:
      - sns: ses-discourse-inbound

# CloudFormation resource templates
resources:
  Resources:
    EmailDispatcherLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        RetentionInDays: "30"

plugins:
  - serverless-python-requirements
custom:
  pythonRequirements:
    pythonBin: .venv/bin/python
    dockerizePip: false

Thx, good start even if it’s not js.

  1. how does one create that (encrypted) API key? Does it need to be encrypted?
  2. Do I need to check “manual polling enabled” Push emails using the API for email replies. in order to get the api endpoint to be active or is it so by default?
  3. Anything to pay extra attention to in terms of AWS permissions or SES rule?

The key doesn’t need to be encrypted. The infrastructure code for our project is in a shared space, so I didn’t want to hard-code the key into the source.

I do have “manual polling enabled” on my site although I don’t remember specifically what that does.

Nothing unusual that I can recall with the AWS permissions beyond sending and read/write to the temporary bucket.

Good luck!

Is there someplace in the webui where one generates the api key. Can’t find any place in the settings. Maybe you just set it to what you want as env var when you build? Anyway no clue how to generate an api key.

The Discourse API key? It is generated in the admin interface at https://{discourse_url}/admin/api/keys

1 Like

Yea I saw that but didn’t think it was right because “granular” had no handle_email permission and was unsure which user or all users to choose.

So I’ve generated a key for user “system” with global permission. Is that what you did?

Well, I was wanting to use my “postman” app to learn/try it out but with no documentation I don’t know what to POST in what format(s) & how to pass the key. If I was more familiar with python I could maybe grok this by looking at your code.

I know this is open source project so it’s understandable there is no real documentation on using the handle_mail api only some help from folks like you…thx!

It’s not undocumented because it’s open source, it’s undocumented because no one (or not many) have asked.

Suggestions that you want to POST the encoded email in an email_encoded thingy. routes.rb shows that it’s a POST and the route, which is https://discourse.example.com/admin/email/handle_mail. I got it from the mail receiver sample template (and I’d recommend that you just use that, as if you had, you’d be finished already) but it’s also in discourse/routes.rb at main · discourse/discourse · GitHub (which isn’t immediately clear just what’s happening unless you understand rals).

1 Like

“mail receiver sample template”
Where is it located? in the repo? GitHub - discourse/discourse: A platform for community discussion. Free, open, simple. ?

I was able to pretty much figure it out from the python code

Success after couple days effort!

@dltj I was able to rewrite your code into nodejs using the aws-sdk and got modules. Thanks so much.

There were serveral road blocks along the way, from AWS (ses,IAM,S3,lambda), messing with serverless, getting the API call correct, to getting the discourse settings proper.

What I am going to do is write a detailed topic on exactly what I did to get this working, enough detail that someone could enable this without having to do any to coding or pulling hairs. The real advantage to this method is that by using ses one does NOT need to set up a email server for the domain. This leaves you free to set up a totally independent mail server for your domain.

If you are interested in how it was done check back I’ll post a link here when it’s ready (a few days).

3 Likes

Adding to this, @dltj we will be removing the email param from the admin/email/handle_mail route soon, you need to send the body of the email as a base64 encoded string in an email_encoded param to this route.

3 Likes

encoding was one of issues I could not resolve
I could not get the encoded email to work

  1. @martin per your post email_encoded is incorret. I’ve tried both, email_encoded throws error, but encoded_email still gives the warning.

body: 'warning: the email parameter is deprecated. all POST requests to this route should be sent with a base64 strict encoded encoded_email parameter instead. email has been received and is queued for processing',

  1. Do I need to convert my incoming mail to plain text (which I am doing now) and back to base64 or is it likely that what AWS Ses stores in in the bucket is already base64? If I pass the email body as SES saved it fails to show up in the received email log. The post command returns no error just the warning. So basically I’m stuck here.

Please delay removing email key until we can get this to work.

even if I decode to plain text and recode to base64 I still get the warning and the email does not show up as received :roll_eyes:

      const en_package = {
      headers: DISCOURSE_API_HEADERS,
      json: {encoded_email: Buffer.from(body.toString(),'base64') }
    }
   
   console.log(Buffer.from(body.toString(),'base64'))
   const res = await rest.post(DISCOURSE_URL,en_package)

once again if I sent plain text with “emal” the same message is received and processed.

You keep saying “strict” does that mean anything? Nodejs/JS has only two base64 flavors “base64” base64url".

Why does api keep giving the warning if I use encoded_email even if I have an error in encoding.

1 Like

I tried curl and is doesnt like encoded_email but it was ok with email_encoded…arrgh so the warning is incorrect!

So again I tried my code but with email_encoded but this time I used a base64 encode module instead of the built in one and it’s working phew!

2 Likes

Ah! I see that now. Thank you for the heads-up, @martin — I’ll add this to my to-do list.

2 Likes

Wow that is totally my bad (messing up the warning), will do a minor PR to fix it.

2 Likes

This PR will fix it, should be merged later today DEV: Fix typo for email encoded by martin-brennan · Pull Request #15577 · discourse/discourse · GitHub.

2 Likes