Troubleshooting spam issue with SendGrid - email notifications arriving in junk folders

email

(Nick Putman) #1

Hi,

I have discourse set up with SendGrid, and most users are experiencing issues with notifications from the forum going into spam/junk folders. I have SPF, DKIM and DMARC records set up on my server.

I think the issue is related to SendGrid, but I’m not sure. I am wondering if there is a way of manually sending emails from discourse to junk/spam checking services such as Is Not Spam, SpamScoreChecker, etc. I suppose I could add new users using the email addresses linked to these services, create a group for these users, and then post content in this group, but I am wondering if there is a simpler way?

I am also wondering about people’s experience of SendGrid and whether I need to be looking to another service?

Thanks,

Nick


(Jeff Atwood) #2

Try using http://mail-tester.com to see.

One thing I found is that sometimes you have to pay for a dedicated IP: if you happen to be sharing that IP with someone who is sending real spam, you are totally screwed.


(Nick Putman) #3

Thanks Jeff. Yes, http://mail-tester.com is a service I have used before.

I have a dedicated IP, with the correct mail settings, which is what leads me to wonder if the issue is related to SendGrid.

I am wondering whether there is a facility within discourse where I can send a test email to Mail Tester, or whether I have to create a user account with the temporary Mail Tester email address?


(Jeff Atwood) #4

Yes, simply go to Admin, Emails, send test email… it’s right there.


(Nick Putman) #5

Thanks - that works well. I got an 8/10 score on Mail Tester.

The two issues are as follows:

-1.084	URIBL_GREY	Contains an URL listed in the URIBL greylist
URIs: sendgrid.net
You can request to be removed from the URIBL greyest

Which would seem to suggest that sendgrid doesn’t have a great reputation.

and…

[Sender ID] forum.open-dialogue.net does not allow your server 167.89.55.59 to use noreply@forum.open-dialogue.net
Sender ID is like SPF, but it checks the FROM address, not the bounce address.
You do not have a SPF record, please add the following one to your domain forum.open-dialogue.net:

v=spf1 a mx include:sendgrid.net ~all

I have added this SPF record to my primary domain open-dialogue.net, but I am not sure how to do so for the subdomain forum.open-dialogue.net.

Do you have any suggestions?


(Matt Palmer) #6

SPF records work exactly the same no matter where in the hierarchy they live. Put an SPF or appropriately-formatted TXT record on forum.open-dialogue.net.


(Nick Putman) #7

Thanks Matt. I have tried to do this in the settings for my domain on Digital Ocean. I have entered the following TXT record under the domain open-dialogue.net:

Name: forum.open-dialogue.net
Text: v=spf1 a mx include:sendgrid.net ~all

But I am still getting the same message on Mail Tester.

I am not sure if this is because I need to wait for my DNS to update, or because I have not entered the settings correctly.

Can you advise further?


(Matt Palmer) #8

I’m not seeing the DNS record in place, and neither is Google:


$ dig forum.open-dialogue.net txt

; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> forum.open-dialogue.net txt
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48159
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;forum.open-dialogue.net.       IN      TXT

;; AUTHORITY SECTION:
open-dialogue.net.      1763    IN      SOA     ns1.digitalocean.com. hostmaster.open-dialogue.net. 1471553742 10800 3600 604800 1800

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Aug 19 19:37:36 AEST 2016
;; MSG SIZE  rcvd: 119

$ dig @8.8.8.8 forum.open-dialogue.net txt

; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> @8.8.8.8 forum.open-dialogue.net txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6223
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;forum.open-dialogue.net.       IN      TXT

;; AUTHORITY SECTION:
open-dialogue.net.      1799    IN      SOA     ns1.digitalocean.com. hostmaster.open-dialogue.net. 1471553742 10800 3600 604800 1800

;; Query time: 311 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Aug 19 19:37:51 AEST 2016
;; MSG SIZE  rcvd: 119

Similarly, a cache-skipping +trace run doesn’t show it, so time to go back and double-check that record, and perhaps get in DO’s ear if you can’t see what’s wrong.


(Nick Putman) #9

I have sorted the issue with the SPF record - I simply had to use ‘forum’ as the name of the TXT record. I am currently getting a score of 8.9/10, with the primary issue being URIBL greylist as mentioned above, which seemingly I can’t do anything about.

There are a couple of other minor issues which don’t seem to affect the score but which have been flagged by Mail Tester, and I’m wondering if I can do anything about these?

Firstly:

Your domains are not aligned. We can't check DMARC.
A DMARC policy allows a sender to indicate that their emails are protected by SPF and/or DKIM, and give instruction if neither of those authentication methods passes. Please be sure you have a DKIM and SPF set before using DMARC.
Before using DMARC, you should make sure the domains used in the Envelope From (e.g., Return-Path or Mail-From), the "Friendly" From (i.e., "Header" From) and the d=domain in the DKIM-Signature are the same
DMARC DNS entry found for the domain _dmarc.forum.open-dialogue.net:

"v=DMARC1; p=none"
Verification details:

mail-tester.com; dmarc=none header.from=forum.open-dialogue.net
mail-tester.com;	dkim=pass (1024-bit key; unprotected) header.d=sendgrid.net header.i=@sendgrid.net header.b=BdpIOUIj;	dkim-atps=neutral
From Domain: forum.open-dialogue.net
DKIM Domain: sendgrid.net

Then:

Weight of the HTML version of your message: 8KB.

Your message contains 22% of text.

(Nick Putman) #10

Also, a fairly simple question - should asking users to add noreply@forum.open-dialogue.net to their safe senders list (and/or marking emails from this address as not junk/spam) be enough to ensure that all emails lands in inboxes?


(Matt Palmer) #11

The only guarantee when it comes to mail delivery is that there are no guarantees. Asking people to whitelist your sending address or doing anything else in their mail client might help, but every mail recipient service treats those signals differently.


(Wes Osborn) #12

Did you enable click tracking in your sendgrid configuration? If so, that is probably what is causing your forum URLs to be re-written as sendgrid.net URLs. Try with the tracking disabled and see if that helps improve your score.


(Felix Freiberger) #13

Actually, there always is a Sendgrid link in the test email – so please do check that you disabled click tracking, but doing so will not eliminate the warning in mail-tester.com.


(Nick Putman) #14

I think that was enabled by default. Now that I have disabled it, it does appear to have removed the URIBL_GREY score. However there are now other issues being reported. The main one affecting the score is this:

-1.729 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)

It seems like RAZOR2_CHECK scores are based on an analysis of the content of the email, rather than having anything to do with the sender - is there any way of addressing this?

There are also a couple of other more minor issues:

-0.276	URIBL_RHS_DOB	Contains an URI of a new domain (Day Old Bread)
URIs: open-dialogue.net

and

Weight of the HTML version of your message: 3KB. Your message contains 63% of text.

and finally

Your message does not contain a List-Unsubscribe header

Is there anything that can be done about any of these issues?


(Jeff Atwood) #15

No, there is not – funnily enough I converted it to a goo.gl redirected link a month or so ago because of this exact issue :wink:


(Matt Palmer) #16

Is that error specifically referring to the test e-mail? If so, that’s fine – we just did a round of “add List-Unsub to everywhere” changes, so as long as you’re running up-to-date code, every real e-mail should have the header set appropriately.


(Felix Freiberger) #17

I even thought of that possibility before posting, and decided not to check :blush:

As a believer of Murphy’s Law, now I’m sure that you hadn’t done that change before I posted, but my posting created a ripple in space-time that allowed you to travel back in time and correct this one month ago :smiley:


(Nick Putman) #18

Yes, its specifically referring to test email - good to know that this isn’t an issue with real emails.


(Nick Putman) #19

Any ideas about this? It wasn’t appearing in spam test reports a couple of days ago, but now is. Are others seeing this entry in spam test checks?