Weird transactional email question


(Andrew Stroup) #1

Sooo … I’ll start off with “I thought I had a good understanding of how to setup SES or Mandrill”, but apparently I’m missing something here.

So to preface my setup is > AWS (EC2, Redis, postgres) > because of the constraints of my organization.


SCENARIO 1:

When setting up the Discourse server I used SES and my @gmail.com account (let’s call it admin@gmail.com for simplicity), which was added using the ‘Verify a New Email Address’ process for individual email addresses.

The SMTP settings within app.yml look like the following:

DISCOURSE_SMTP_ADDRESS: email-smtp.us-east-1.amazonaws.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: [AWS SES USERNAME]
DISCOURSE_SMTP_PASSWORD: [AWS SES PASSWORD]

Using the admin@gmail.com account, I am able to successfully:

  • send an email to other @gmail.com email addresses
  • send emails to my @custom.com domain email addresses

I then use my @custom.com (successfully verified by SES) with Discourse and here’s what happens:

  • I can send emails to @gmail.com accounts
  • BUT not to @custom.com email addresses (including the one I’m sending it with).

NOTE: Neither @gmail.com or @custom.com are verified domains within SES.


SCENARIO 2:

My organization also has a Mandrill account used across the organization by several different domains (successfully). I have done the following:

  • added @custom.com to the list of sending domains
  • have verified the domain and DKIM but receive an SPIF error because it’s not a top 10 record)
  • created an API key and added the updated SMTP info into the app.yml file, which looks like the following

DISCOURSE_SMTP_ADDRESS: smtp.mandrillapp.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: [Mandrill user email address]
DISCOURSE_SMTP_PASSWORD: [Mandrill API key]

When attempting to send emails via @custom.com while using Mandrill I can’t send an email from Discourse to @custom.com OR @gmail.com email addresses, BUT I receive confirmed delivery via the Mandrill API logs …

Method: messages/send-raw.json
Time: May 11, 2015 3:47 pm
Call time: 69.4ms
IP: [Discourse server EC2 elastic IP address]
User Agent: Mandrill-Python/1.0.56

Full Request

{
    "send_at": null,
    "from_name": null,
    "raw_message": "Received: from localhost.localdomain (unknown [EC2 Elastic IP address])\n\t(Authenticated sender: MI0xXUE4xIA5TTfSyziQcg@custom.com)\n\tby ip-10-187-29-39 (Postfix) with ESMTPSA id 2AC4DC0736\n\tfor <admin@custom.com>; Mon, 11 May 2015 19:47:08 +0000 (UTC)\nDate: Mon, 11 May 2015 19:47:37 +0000\nFrom: admin@custom.com\nReply-To: admin@custom.com\nTo: admin@custom.com\nMessage-ID: <8d5c1004-4d64-4722-8f50-5fc1cde5c535@discourse.custom.com>\nSubject: [Custom Discourse] Email Deliverability Test\nMime-Version: 1.0\nContent-Type: multipart/alternative;\n boundary=\"--==_mimepart_55510759a2805_6a3ff1b5cd7e7c285b2\";\n charset=UTF-8\nContent-Transfer-Encoding: 7bit\nAuto-Submitted: auto-generated\n\n\n----==_mimepart_55510759a2805_6a3ff1b5cd7e7c285b2\nContent-Type: text/plain;\n charset=UTF-8\nContent-Transfer-Encoding: 7bit\n\nThis is a test email from\n\n[**http://discourse.custom.com**][0]\n\nEmail deliverability is complicated. Here are .."
    "ip_pool": null,
    "from_email": null,
    "return_path_domain": null,
    "to": [
        "admin@gmail.com"
    ],
    "key": "MI0xXUE4xIA5TTfSyziQcg",
    "async": false
}

Full Response

[
    {
        "email": "admin@gmail.com",
        "status": "sent",
        "_id": "2312b8907c184e1985e046c1c2e57f0a",
        "reject_reason": null
    }
]

So there’s obviously multiple problems going on here, which from what I understand is …

  • there’s something preventing me from delivering an email from @custom.com to my own (or other) @custom.com email addresses via SES
  • I’m completely unable to successfully deliver any emails (to @gmail.com or @custom.com) from Mandrill, even with a domain and DKIM verified and SPF record added (not verified due to not Top 10) for @custom.com

The only thing I can think of is that in the “raw_message” there’s the following …

“Received: from localhost.localdomain (unknown [Discourse EC2 elastic IP address])\n…”

where the actual EC2 elastic IP address is used. Could this be the cause and any idea on how to resolve?

The other thought is that the @custom.com spam or whitelist filter is blocking me from successfully delivering emails, BUT this doesn’t make sense because I’m using a verified email from the same exact @custom.com domain.

Thoughts? Thanks so much!


(Dean Taylor) #2

###SCENARIO 1

My guess is SPF records for @custom.com did not allow Amazon SES to send mail.

###SCENARIO 2

Well it’s not a valid SPF record then - you never want errors on SPF records - crazy unpredictability for mail delivery.

SPF specification has a limit on the number of DNS lookups (10) required to fully resolve an SPF record, this is overed in detail here:
http://www.openspf.org/RFC_4408#processing-limits

You can check this with this tool:
http://www.kitterman.com/spf/validate.html

Here you have contradicted yourself - now you are saying they are verified??

###General mail testing
The following tool has been mentioned a few times on this forum - it might help you:

In addition any test messages you send should look like real messages - never send a message with “Test” as the subject and “Test” as the body. It will likely never get delivered, full sentences, multiple lines - make it look real.


(Andrew Stroup) #3

Mean to include that the SPF still has an error because it’s not a top 10 record (modified original post)!

Working on the other things you mentioned …

  1. SPF Query Tool results

I totally get what you’re saying about the SPF record not being valid, BUT I just don’t think that’s the issue, specifically because I can send emails to whoever from my @gmail.com account in SES. If I needed a valid SPF record to do this …

  • other accounts within Mandrill that don’t have either a verified domain, DKIM or SPF record shouldn’t be able to send emails, BUT they can send emails out just fine
  • I shouldn’t be able to send out emails from my @gmail.com account within SES

Unless I’m missing something, appreciate all the help!


(Dean Taylor) #4

the @gmail.com domain has a softfail SPF record:

C:\Users\Dean>dig @8.8.8.8 gmail.com ANY

; <<>> DiG 9.9.2-P1 <<>> @8.8.8.8 gmail.com ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34817
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;gmail.com.                     IN      ANY

;; ANSWER SECTION:
gmail.com.              299     IN      A       216.58.208.69
gmail.com.              299     IN      AAAA    2a00:1450:4009:80a::2005
gmail.com.              21599   IN      NS      ns2.google.com.
gmail.com.              3599    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.              21599   IN      NS      ns3.google.com.
gmail.com.              21599   IN      NS      ns1.google.com.
gmail.com.              3599    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
gmail.com.              3599    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.              21599   IN      NS      ns4.google.com.
gmail.com.              3599    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.              299     IN      TXT     "v=spf1 redirect=_spf.google.com"
gmail.com.              21599   IN      SOA     ns1.google.com. dns-admin.google.com. 2015031901 216
00 3600 1209600 300
gmail.com.              3599    IN      MX      10 alt1.gmail-smtp-in.l.google.com.

;; Query time: 27 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu May 14 00:18:03 2015
;; MSG SIZE  rcvd: 367


C:\Users\Dean>dig @8.8.8.8 _spf.google.com ANY

; <<>> DiG 9.9.2-P1 <<>> @8.8.8.8 _spf.google.com ANY
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61411
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_spf.google.com.               IN      ANY

;; ANSWER SECTION:
_spf.google.com.        299     IN      TXT     "v=spf1 include:_netblocks.google.com include:_netbl
ocks2.google.com include:_netblocks3.google.com ~all"

;; Query time: 30 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu May 14 00:25:43 2015
;; MSG SIZE  rcvd: 160

Because it has a soft fail (~all) SPF record - it isn’t used to prevent the delivery of emails from *@gmail.com email addresses, treated as accept but mark.

Your domain does have an SPF record, worse than that it has an invalid SPF record - this is used to prevent the delivery of mail.

If you removed the SPF record for your domain - it might deliver - but you would have to wait for the DNS TTL time to expire before attempting another test delivery.

However I don’t believe this will help you - and SPF record will be required as AWS IP addresses are commonly blocked (read: negatively effected) because they are used by spammers; adding an (correct) SPF record works around this.


(Andrew Stroup) #5

Hm got it, thanks Dean, standby for more, working this out now …