Configuring AWS SES for outgoing, bounce, and incoming email

Community wiki Sysadmins
,
1

I thought I’d share the configuration I came up with to use AWS SES for outgoing, bounce, and incoming email. There’s definitely some nuance to the SES service, and it took a good deal of trial and error to understand exactly how it works. This is more of a brain-dump than step-by-step-follow-the-dotted-line. It should be unnecessary, but use at your own risk. And by all means always read through and understand any code written by others you implement.

Background:

I’m working to deploy Discourse in AWS and utilize all their services I can to ensure reliability and redundancy. As a developer I’m more comfortable with the command line and code, and wanted to use IaC automation. My whole environment is being deployed with Terraform, but I’ve tried to click through the web console and line things up as best I can. IAM and policy documents are beyond the scope of this, but I believe I’ve called out where things are needed.

Running a Postfix instance seems like over kill for a single application. Using a POP3 mailbox is so very 90’s. So down the AWS rabbit hole I went.

I did find some extremely useful posts which aided my quest

The mail-receiver container also helped me understand how Discourse digests messages

Initially I expected the AWS webhook endpoint to handle incoming messages, but after going though the code realized it wouldn’t. I based my lambda receiver code on the excellent example by @dltj. I opted to use SNS for message delivery instead of S3.

Prereqs

  • AWS Account
  • Working knowledge of DNS and the email related record types
  • A domain (or subdomain) in which you can make changes

Notes

  • Everything documented must be created in the same AWS region
  • Bold italicized text like this are your implementation specific values
  • Italicized text are names of variables, fixed values, or UI elements

Steps

  1. Create a Simple Email Service (SES) domain identity, your.domain, in one of the AWS regions supporting email receiving

  2. Verify domain identity

  3. Create a Simple Notification Service (SNS) topic, feedback-sns-topic, for feedback notifications

  4. Configure the your.domain domain identity
    a. Enable email feedback forwarding
    b. Configure bounce and complaint (not delivery) feedback notifications to use SNS feedback-sns-topic topic

  5. Create a subscription on the SNS feedback-sns-topic topic
    a. Protocol is HTTPS (you’re not still using HTTP are you?)
    b. Set endpoint to https://your.domain/webhooks/aws (see VERP post)
    c. Select enable raw message delivery

  6. Create another SNS topic, incoming-sns-topic, for incoming email

  7. Create an SES email receiving rule set, inbound-mail-set, if there isn’t an existing active one. If so use that as there can only be one active rule set

  8. Create a receipt rule in the inbound-mail-set receiving rule set
    a. Set recipient condition to your.domain
    b. Add action to publish to SNS topic incoming-sns-topic, encoding Base64

  9. Create API key in your Discourse instance for user system, granting receive email action on the email resource

  10. Create a secret in Secret Manager, email-handler-secret, with the following keys and their respective values:

    • api_endpoint - https://your.domain/admin/email/handle_mail
    • api_key - from step 9
    • api_username - system, unless you used something different in step 9

  11. Create a Lambda layer, lambda-receiver-layer, for the python3.10 runtime containing the requests and aws-lambda-powertools libraries

  12. Create a lambda function, email-receiver-lambda, for the python3.10 runtime with the receiver code:

# Copyright (c) 2023 Derek J. Lambert
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
import json
import os
from typing import TypedDict

import requests
from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.data_classes import event_source
from aws_lambda_powertools.utilities.data_classes.sns_event import SNSEvent, SNSEventRecord
from aws_lambda_powertools.utilities.typing import LambdaContext


class Secret(TypedDict):
    api_endpoint: str
    api_username: str
    api_key: str


service = os.getenv('AWS_LAMBDA_FUNCTION_NAME')
logger  = Logger(log_uncaught_exceptions=True, service=service)

try:
    SECRET_NAME = os.environ['SECRET_NAME']
except KeyError as e:
    raise RuntimeError(f'Missing {e} environment variable')

AWS_EXTENSION_PORT = os.getenv('PARAMETERS_SECRETS_EXTENSION_HTTP_PORT', 2773)
EXTENSION_ENDPOINT = f'http://localhost:{AWS_EXTENSION_PORT}/secretsmanager/get?secretId={SECRET_NAME}'


def get_secret() -> Secret:
    return parameters.get_secret(SECRET_NAME, transform='json')


def handle_record(record: SNSEventRecord):
    sns         = record.sns
    sns_message = json.loads(sns.message)

    try:
        message_type    = sns_message['notificationType']
        message_mail    = sns_message['mail']
        message_content = sns_message['content']
        message_receipt = sns_message['receipt']
    except KeyError as exc:
        raise RuntimeError(f'Key {exc} missing from message')

    try:
        receipt_action = message_receipt['action']
    except KeyError as exc:
        raise RuntimeError(f'Key {exc} missing from receipt')

    try:
        action_encoding = receipt_action['encoding']
    except KeyError as exc:
        raise RuntimeError(f'Key {exc} missing from action')

    try:
        mail_source      = message_mail['source']
        mail_destination = ','.join(message_mail['destination'])
    except KeyError as exc:
        raise RuntimeError(f'Key {exc} missing from mail')

    logger.info(f'Processing SNS {message_type} {sns.get_type} record with MessageId {sns.message_id} from {mail_source} to {mail_destination}')

    # 'email' is deprecated, but just in case something is configured incorrectly
    body_key = 'email_encoded' if action_encoding == 'BASE64' else 'email'

    request_body = {
        body_key: message_content
    }

    secret  = get_secret()
    headers = {
        'Api-Username': secret['api_username'],
        'Api-Key':      secret['api_key'],
    }

    response = requests.post(url=secret['api_endpoint'], headers=headers, json=request_body)

    logger.info(response.text)
    response.raise_for_status()


@event_source(data_class=SNSEvent)
@logger.inject_lambda_context
def lambda_handler(event: SNSEvent, context: LambdaContext):
    for record in event.records:
        handle_record(record)

  1. Configure email-receiver-lambda lambda function:
    a. Add layer lambda-receiver-layer
    b. Add region-specific layer for AWS Parameter Store
    c. Add environment variable SECRET_NAME with the value email-handler-secret
    d. If you’d like additional details logged, add environment variable POWERTOOLS_LOGGER_LOG_EVENT with value true

  2. Grant lambda function email-receiver-lambda IAM permission secretsmanager:GetSecretValue for secret email-handler-secret

  3. Create a subscription on the SNS topic incoming-sns-topic
    a. Protocol is AWS Lambda
    b. Set endpoint to ARN of email-receiver-lambda

  4. IAM permissions will be needed for the SNS subscription on incoming-sns-topic topic to invoke email-receiver-lambda, but I believe this will be done automatically when configured through the console

For debugging purposes, or general self annoyance, you can add an email subscription to either of the SNS topics to monitor the notifications.

I put this down in a couple sittings, but I think it’s everything. I can try and answer general questions as time permits.

8 Likes
2

Updates to original post

Version 2

# Copyright (c) 2023 Derek J. Lambert
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
import os
from enum import Enum
from typing import Literal, Optional

import requests
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import utils
from aws_lambda_powertools.utilities.parser import BaseModel, event_parser
from aws_lambda_powertools.utilities.parser.models import SnsModel, SesMessage, SnsRecordModel, SesMail, SesReceipt, SesMailCommonHeaders
from aws_lambda_powertools.utilities.typing import LambdaContext


class Secret(BaseModel):
    api_endpoint: str
    api_username: str
    api_key:      str


class SnsSesActionEncoding(str, Enum):
    BASE64 = 'BASE64'
    UTF8   = 'UTF8'


class SnsSesReceiptAction(BaseModel):
    type:     Literal['SNS']
    encoding: SnsSesActionEncoding
    topicArn: str


class SnsSesReceipt(SesReceipt):
    action: SnsSesReceiptAction


class SnsSesMailCommonHeaders(SesMailCommonHeaders):
    returnPath: Optional[str]


class SnsSesMail(SesMail):
    commonHeaders: SnsSesMailCommonHeaders


class SnsSesMessage(SesMessage):
    notificationType: str  # TODO: Are there other values besides 'Received'?
    content:          str
    mail:             SnsSesMail
    receipt:          SnsSesReceipt


try:
    SECRET_NAME       = os.environ['SECRET_NAME']
    AWS_SESSION_TOKEN = os.environ['AWS_SESSION_TOKEN']
except KeyError as e:
    raise RuntimeError(f'Missing {e} environment variable')

AWS_EXTENSION_PORT = os.getenv('PARAMETERS_SECRETS_EXTENSION_HTTP_PORT', 2773)

logger = Logger(service=os.getenv('AWS_LAMBDA_FUNCTION_NAME'), log_uncaught_exceptions=True, use_rfc3339=True)

utils.copy_config_to_registered_loggers(source_logger=logger)


def get_secret() -> Secret:
    # AWS Parameters and Secrets Lambda Extension
    # https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html

    response = requests.get(
        url=f'http://localhost:{AWS_EXTENSION_PORT}/secretsmanager/get?secretId={SECRET_NAME}',
        headers={
            'X-Aws-Parameters-Secrets-Token': AWS_SESSION_TOKEN
        }
    )

    try:
        response.raise_for_status()
    except Exception:
        logger.critical(response.text)
        raise

    return Secret.parse_raw(response.json()['SecretString'])


def handle_record(record: SnsRecordModel):
    sns_record       = record.Sns
    sns_ses_message  = SnsSesMessage.parse_raw(record.Sns.Message)
    mail_destination = ','.join(sns_ses_message.mail.destination)

    logger.info(f'Processing SNS {sns_ses_message.notificationType} notification record with MessageId {sns_record.MessageId} from {sns_ses_message.mail.source} to {mail_destination}')

    # 'email' is deprecated, but just in case something is configured incorrectly
    body_key = 'email_encoded' if sns_ses_message.receipt.action.encoding is SnsSesActionEncoding.BASE64 else 'email'
    secret   = get_secret()

    response = requests.post(
        url=secret.api_endpoint,
        headers={
            'Api-Username': secret.api_username,
            'Api-Key':      secret.api_key,
        },
        json={
            body_key: sns_ses_message.content
        }
    )

    try:
        response.raise_for_status()
    except Exception:
        logger.critical(response.text)
        raise

    logger.info(f'Endpoint response: {response.text}')


@event_parser(model=SnsModel)
@logger.inject_lambda_context
def lambda_handler(event: SnsModel, context: LambdaContext):
    for record in event.Records:
        handle_record(record)
1 Like
3

Thanks for this guide @dlambert :smiley:

I was doing great, until I got to step 11:

Where / how do I create this? :thinking:

4

Do you get it working?

I also get stuck at step 11. don’t know what to do next. anyone could help?

Screenshot 2023-11-28 021900
Screenshot 2023-11-28 021900898×799 38.1 KB

1 Like
5

No, sorry, I gave up and we disabled all reply-by-email functionality, using SES for simple outbound email only :cry:

6

I tried to follow all the steps to setup but in the end, I am getting this error in Cloudwatch can anyone help with this?

[ERROR] HTTPError: 403 Client Error: Forbidden for url: https://forum.siteurl.com/admin/email/handle_mail
Traceback (most recent call last):
  File "/opt/python/aws_lambda_powertools/middleware_factory/factory.py", line 135, in wrapper
    response = middleware()
  File "/opt/python/aws_lambda_powertools/utilities/data_classes/event_source.py", line 39, in event_source
    return handler(data_class(event), context)
  File "/opt/python/aws_lambda_powertools/logging/logger.py", line 453, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/task/lambda_function.py", line 107, in lambda_handler
    handle_record(record)
  File "/var/task/lambda_function.py", line 100, in handle_record
    response.raise_for_status()
  File "/opt/python/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
7

Okay, it was because of Cloudflare disabling resolved the issue. maybe later on I’ll write here how I made it work following all the steps. :slight_smile:

1 Like
8

This is what I did.

Installed Python 3.10 on my PC, after step 10.

Then run these commands.

mkdir lambda-receiver-layer

cd lambda-receiver-layer

mkdir python

pip install requests aws-lambda-powertools -t ./python

touch ./python/__init__.py

As I had issues with urllib3

Here are additional steps so you don’t get that error.

In your lambda-receiver-layer directory create this file requirements.txt

add the following line in this file requirements.txt:

urllib3<2

Then Run the following command

pip install -r requirements.txt -t layer

Now another folder will be created inside lambda-receiver-layer directory named layer

Copy all the contents of layer to python folder

Now, right-click on the Python folder and click ‘Compress to ZIP’ rename this zip to lambda-receiver-layer

Now, Go back to the AWS Management Console, go to the Lambda service, and navigate to “Layers.” Click on “Create Layer,” put this in the name lambda-receiver-layer and upload the zip archive you created. In runtime add Python 3.10 then click create.

Now follow back from step 12 from the original post.

9

I’m getting stuck at step 11, where do I paste the python code?

10

I need urgent help to fix my SMTP Bounces in several instances, I’ve posted a marketplace job Fix AWS SNS Bounce

11

I’m stuck at point 14, anybody can clarify what I’ve to do?