Handling bouncing e-mails


(Régis Hanol) #84

Sure, I’ll have a look.


(Régis Hanol) #85

Amazon SNS Notification webhook has been added :tada:

It will also automatically confirm the SNS topic subscription.


(Michał Frąckiewicz) #86

Awesome, thanks! :heart:


(Michał Frąckiewicz) #87

I switched to AWS SES on v2.3.0.beta3 for a few days but bounces webhooks aren’t handled properly.
I sent test email using /admin/email to nonexistent mailbox but no bounce appeared on admin panel even after I wait few minutes (webhooks are almost instant).

I also subscribed to this same bounces topic via email too so I see that webhooks are generated properly, example:

{
  "Type" : "Notification",
  "MessageId" : "7ca3dceb-ae85-553f-979b-redacted",
  "TopicArn" : "arn:aws:sns:eu-west-1:redacted",
  "Message" : "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"testytesttest@redacted\",\"action\":\"failed\",\"status\":\"5.1.1\",\"diagnosticCode\":\"smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try\\n550-5.1.1 double-checking the recipient's email address for typos or\\n550-5.1.1 unnecessary spaces. Learn more at\\n550 5.1.1  https://support.google.com/mail/?p=NoSuchUser w14si978073ede.350 - gsmtp\"}],\"timestamp\":\"2019-03-18T18:41:20.982Z\",\"feedbackId\":\"01020169921bac19-bb89b0ef-47f3-42cd-b545-98442b03428c-000000\",\"remoteMtaIp\":\"74.125.193.27\",\"reportingMTA\":\"dsn; a7-13.smtp-out.eu-west-1.amazonses.com\"},\"mail\":{\"timestamp\":\"2019-03-18T18:41:20.000Z\",\"source\":\"noreply@redacted\",\"sourceArn\":\"arn:aws:ses:eu-west-1:redacted\",\"sourceIp\":\"redacted\",\"sendingAccountId\":\"810186810848\",\"messageId\":\"01020169921baac7-f3f11807-849f-4025-8887-e6ea3ec76bcd-000000\",\"destination\":[\"testytesttest@redacted\"]}}",
  "Timestamp" : "2019-03-18T18:41:20.997Z",
  "SignatureVersion" : "1",
  "Signature" : "redacted",
  "SigningCertURL" : "redacted",
  "UnsubscribeURL" : "redacted"
}


Maybe I set something wrong? Subscription to example.com/webhooks/aws is successful in AWS panel, bounces are visible in http logs:

[18/Mar/2019:18:41:21 +0000] "example.com" 54.240.197.98 "POST /webhooks/aws HTTP/1.1" "Amazon Simple Notification Service Agent" "-" 200 796 "-" 0.008 0.008 "-"
[18/Mar/2019:18:38:06 +0000] "example.com" 54.240.197.106 "POST /webhooks/aws HTTP/1.1" "Amazon Simple Notification Service Agent" "-" 200 796 "-" 0.004 0.011 "-"
[18/Mar/2019:18:36:51 +0000] "example.com" 54.240.197.71 "POST /webhooks/aws HTTP/1.1" "Amazon Simple Notification Service Agent" "-" 200 796 "-" 0.016 0.014 "-"

production.log :

Started POST "/webhooks/aws" for 54.240.197.71 at 2019-03-18 18:36:51 +0000
Processing by WebhooksController#aws as HTML
  Rendering text template
  Rendered text template (0.0ms)
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms)

production_errors.log is empty

Nothing seems wrong but it doesn’t work, any tips how I should debug this further?


(Régis Hanol) #88

Can you ssh into your server and then run

cd /var/discourse
./launcher enter app
rails c

And then do this line by line and tell me which one failed?

raw  = "<the webhook content>"
json = JSON.parse(raw)

message = json["Message"].presence
message["notificationType"] == "Bounce"
message_id = message.dig("mail", "messageId").presence
bounce_type = message.dig("bounce", "bounceType").presence

require "aws-sdk-sns"
Aws::SNS::MessageVerifier.new.authentic?(raw)

(Michał Frąckiewicz) #89

I guess it shouldn’t be this way :stuck_out_tongue:

[17] pry(main)> message["notificationType"] == "Bounce"
=> false

(Régis Hanol) #90

Weird, what’s the value of message["notificationType"]?


(Michał Frąckiewicz) #91
[26] pry(main)> message["notificationType"]
=> "notificationType"

(Régis Hanol) #92

That’s even weirder :thinking:… Can you show message?


(Michał Frąckiewicz) #93

Sorry for one line, however should be readable in Discourse history/edit mode

[30] pry(main)> message
=> "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"testytesttest@example.com\",\"action\":\"failed\",\"status\":\"5.1.1\",\"diagnosticCode\":\"smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try\\n550-5.1.1 double-checking the recipient's email address for typos or\\n550-5.1.1 unnecessary spaces. Learn more at\\n550 5.1.1  https://support.google.com/mail/?p=NoSuchUser w14si978073ede.350 - gsmtp\"}],\"timestamp\":\"2019-03-18T18:41:20.982Z\",\"feedbackId\":\"01020169921bac19-bb89b0ef-47f3-42cd-b545-98442b03428c-000000\",\"remoteMtaIp\":\"74.125.193.27\",\"reportingMTA\":\"dsn; a7-13.smtp-out.eu-west-1.amazonses.com\"},\"mail\":{\"timestamp\":\"2019-03-18T18:41:20.000Z\",\"source\":\"noreply@example.com\",\"sourceArn\":\"redacted\",\"sourceIp\":\"redacted\",\"sendingAccountId\":\"redacted\",\"messageId\":\"01020169921baac7-f3f11807-849f-4025-8887-e6ea3ec76bcd-000000\",\"destination\":[\"testytesttest@example.com\"]}}"

(Régis Hanol) #94

Haha. I know what’s the issue. :tada: Lemme fix :muscle:


(Régis Hanol) #95

Ok, that should fix it


(Michał Frąckiewicz) #96

Yep, it should work now, I’ll wait for beta6 to check it :slight_smile:

[33] pry(main)> message["notificationType"] == "Bounce"
=> true

(Régis Hanol) #97

Might be in a little while (like a couple of weeks).


(Michał Frąckiewicz) #98

Thanks for info :slight_smile:
Can I use specific commit of git repo in app.yml container declaration?
Then I can test it almost instantly.


(Jay Pfaffman) #99

If you’re on the tests-passed branch, you can just rebuild or do an upgrade from the web interface and upgrade now.


(Michał Frąckiewicz) #100

I tested fix and it still doesn’t work :frowning:

I debugged this further and everything works except searching for proper message via EmailLog.
No ID in JSON webhook matches ID in EmailLog so this line cannot work:

Dirty fix would be searching for latest message sent to email address that was reported as a bounce, like this:

EmailLog.order("created_at DESC").where(to_address: r["emailAddress"])

Proper fix would set correct message ID when sending, via SMTP, this can be hard depending on current implementation.