Troubleshooting email on a new Discourse install

PackElend · December 4, 2019, 5:44pm

thanks that got me back on track informational wise
I would suggest to extend the comment by saying that this email can be changed after the installation by going to {discourse-domain/admin/site_settings/category/required
(but saying that the SMTP log- in can only be changed by a rebuild, they are read-only in the settings).
How to configure Mailgun Email Server on Discourse Forum correct way? - Detailed and Verified Steps • Crunchify is illustrating the relationships not too bad.
I don’t mind doing the PR, if it could be accepted as an improvement.

I find some good guides for the telnet work, explaining it well what has to be done when TLS is used (second covers Zoho Mail, what I currently use).

and good explanation for What is the difference between the HELO/EHLO commands? | RedinSkala

phil_c64 · December 9, 2019, 10:02pm

Thank you so much for posting this. Was driving me mad not getting email working. DNS records etc etc when all that was needed was:

DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: none

PackElend · December 22, 2019, 12:33am

so I’m done, took me a while so now it comes.
I would say production.log needs some update on how entries in there are populated
I was able to reproduce what Online SMTP tool - Test your Mail Server on the fly does or tells me, respectively using OpenSSL but I had to overcome some obstacles.

My list of obstacles:

verify username and password that they don’t begin with “Q” or “R”, using Base64 Encode - Online Tool or echo -n *text2encode* | base64 as described in ssl - Why do I get DONE after AUTH LOGIN command? - Stack Overflow
use ELHO instead of HELO (or do it the right order) to enable secured connection, as described in
ensure, that you use all fields to be RFC 5322 compliant, as shown on the googles logged answer in sender, subject, recipient EMPTY - hMailServer forum. It should make a difference, what information are in which field, as e.g. described in What is the difference between the “From” and “Sender”? – Mailgun Help Center but some SMPT services don’t use all fields, see SMPT test for details.
add the -crlf arg to openssl s_client otherwise the end of line command could not be interpreted correctly. I canot remember what made me use it but it could the discussion in Question about <CRLF>.CRLF> sequence. - hMailServer forum plus the arguments of openssl s_client listed on /docs/man1.0.2/man1/s_client.html
add the -ign_eof arg to openssl s_client to make here documents work properly as described in the answer of bash - Openssl s_client and HERE documents - Stack Overflow, shell - OpenSSL Script to run checks in bash - Stack Overflow (and some hints in Re: Scripting of s_client, indirectly)

Before I continue, a little excurse first. Can’t remember the reason but I was irritated by the pro and cons of Port 465 and 587. During my research, I came across Sending email failed with SMTPS port 465 what refers to RFC 8314 - Cleartext Considered Obsolete: Use of Transport Layer Security (TLS) for Email Submission and Access what says that port 465 should be the new standard port by using “Implicit TLS” as improper configured STARTTLS can lead to plain-text communication. More information are given in:

SSL, TLS, and STARTTLS | Fastmail
tls - Why are common services using implicit SSL not considered obsolete in the way that SMTPS is? - Information Security Stack Exchange
Thus I began testing if standard cryptography can be enforced by playing around with OpenSSL. Good guidance is given in:
OpenSSL with Bash » Linux Magazine
The Most Common OpenSSL Commands
OpenSSL Cookbook: Chapter 2. Testing with OpenSSL
This references and the solution of the obstacles above led to my standard OpenSSL command:
echo "x" | timeout 10 openssl s_client -connect smtp.eu.mailgun.org:587 -starttls smtp -tls1_2 -crlf -ign_eof

At first, I tested the communication manually.
The guidance was given to me by

Here is an example for mailgun
S…what I typed in the console
R…what the console returned

converting username into a base64 encoded string

S: echo -n test@mailgun.mydomain.com | base64
R: dGVzdEBtYWlsZ3VuLm15ZG9tYWluLmNvbQ==

converting password into a base64 encoded string

S: echo -n mypassword | base64
R: bXlwYXNzd29yZA==

establishing OpenSSL including an automatic timeout to quit the connection after 30 seconds if nothing happens.
timeout 30 o openssl s_client -connect smtp.eu.mailgun.org:587 -starttls smtp -tls1_2 -crlf -ign_eof

sending EHLO and AUTH LOGIN (encrypted login)

S: EHLO test@mailgun.mydomain.com
R: 250-ak47
R: 250-AUTH PLAIN LOGIN
R: 250-SIZE 52428800
R: 250-8BITMIME
R: 250-ENHANCEDSTATUSCODES
R: 250 SMTPUTF8
S: AUTH LOGIN
R: 334 VXNlcm5hbWU6

do the log-in and define the sender and recipient
VXNlcm5hbWU6 decoded means Username
UGFzc3dvcmQ6decoded means Password
You can decode it by the above-mentioend website or do

S: base64 -d
S: VXNlcm5hbWU6
S: *press the key CTRL+d*
R: Username:base64: invalid input

the text before :base 64 is the translation.
So let get ahead with the log-in:

S: AUTH LOGIN
R: 334 VXNlcm5hbWU6
S: dGVzdEBtYWlsZ3VuLm15ZG9tYWluLmNvbQ==
R: 334 UGFzc3dvcmQ6
S: bXlwYXNzd29yZA==
R: 235 2.0.0 OK
S: mail from:<SENDER@domain.com>
R: 250 Sender address accepted
S: rcpt to:<receiver@gmail.com>
R: 250 Recipient address accepted

create the email

S: DATA
R: 354 Continue
S: from: test_sender@mailgun.test.ga
S: to: RECIPIENT@domain.com
S: subject: OpenSSL Mailgun Test
S: this is the email body
S: .
R: 250 Great success
S: quit
R: 221 See you later. Yours truly, Mailgun

Quickly after my first tests, I got sick and tired of writing again and again. Luckily I found out about the HERE document:

So I created a HERE document for each smtp service:

mailgun

openssl s_client -connect smtp.eu.mailgun.org:587 -starttls smtp -tls1_2 -crlf -ign_eof <<EOF 
EHLO test@mailgun.mydomain.com
AUTH LOGIN
dGVzdEBtYWlsZ3VuLm15ZG9tYWluLmNvbQ==
bXlwYXNzd29yZA==
mail from:<SENDER@domain.com>
rcpt to:<MyPrivateEmail@gmail.com>
DATA
from: test_sender@somedomain.ga
to: RECIPIENT@domain.com
subject: OpenSSL Siteground Mail
Mime-Version: 1.0;
Content-Type: text/html; charset="ISO-8859-1";
Content-Transfer-Encoding: 7bit;

<html>
	<body>
		 <ul>
			<li>
				openssl s_client -connect smtp.eu.mailgun.org:587 -starttls smtp -tls1_2 -crlf -ign_eof	
			</li>
			<li>
				EHLO test@mailgun.mydomain.com
			</li>
			<li>
				mail from:&ltSENDER@domain.com&gt
			</li>
			<li>
				rcpt to:&ltMyPrivateEmail@gmail.com&gt
			</li>
			<li>
				from: test_sender@somedomain.ga
			</li>
			<li>
				to: RECIPIENT@domain.com
			</li>

		</ul> 	
	</body>
</html>
.
quit
EOF

Here I figured out that it accpets pretty much an sender & receipent as your are registerd by your credit card

siteground

openssl s_client -connect itm13.siteground.biz:587 -starttls smtp -tls1_2 -crlf -ign_eof <<EOF
EHLO MyEmailAccount@SitegroundHostedDomain.com
AUTH LOGIN
TXlFbWFpbEFjY291bnRAU2l0ZWdyb3VuZEhvc3RlZERvbWFpbi5jb20=
bXlwYXNzd29yZA==
mail from:AnySender@SitegroundHostedDomain.com
rcpt to:MyPrivateEmail@gmail.com
DATA
from: SENDER
to: RECIPIENT
subject: OpenSSL Siteground Mail
Mime-Version: 1.0;
Content-Type: text/html; charset=“ISO-8859-1”;
Content-Transfer-Encoding: 7bit;

openssl s_client -connect smtp.zoho.eu:587 -starttls smtp -tls1_2 -crlf -ign_eof
mail from:&ltAnySender@SitegroundHostedDomain.com&gt
rcpt to:&ltMyPrivateEmail@gmail.com&gt
from: SENDER
to: RECIPIENT

	</ul> 	
</body>

. quit EOF

Here I figured out that you have to use a domain hosted on siteground or you get

siteground error

550-Sorry 157.245.67.230. The “MAIL FROM” email address
550-AnySender@NotRegisteredSubdomain.SitegroundHostedDomain.com / NotRegisteredSubdomain.SitegroundHostedDomain.com you are using is not
550-hosted on this server. You authenticated as
550-MyEmailAccount@SitegroundHostedDomain.com but tried to use
550-AnySender@NotRegisteredSubdomain.SitegroundHostedDomain.com / NotRegisteredSubdomain.SitegroundHostedDomain.com as mail from address.
550-Sending mails from addresses not hosted on our server is not allowed.
550-Please check the “MAIL FROM” settings of your mail client and make sure you
550-set the “MAIL FROM” address to an email account which is hosted on the
550 server.

zoho

openssl s_client -connect smtp.zoho.eu:587 -starttls smtp -tls1_2 -crlf -ign_eof <<EOF
EHLO Discourse@zoho.MyDomain.com
AUTH LOGIN
ZGlzY291cnNlQHpvaG8uTXlEb21haW4uY29t
bXlwYXNzd29yZA==
mail from:
rcpt to:MyPrivateEmail@gmail.com
DATA
from: discourse@zoho.fairbnb.community
to: RECIPIENT
subject: OpenSSL Zoho Mail
Mime-Version: 1.0;
Content-Type: text/html; charset=“ISO-8859-1”;
Content-Transfer-Encoding: 7bit;

openssl s_client -connect smtp.zoho.eu:587 -starttls smtp -tls1_2 -crlf -ign_eof
EHLO Discourse@zoho.MyDomain.com
mail from:&ltDiscourse@zoho.MyDomain.com&gt
rcpt to:&ltMyPrivateEmail@gmail.com&gt
from: Discourse@zoho.MyDomain.com
to: RECIPIENT

	</ul> 	
</body>

.

EOF

Here I figured out that if:

to: does not match rcpt to: recipient is put in BBC and to is ignored
from: does not match Username: sending is refused and you get
553 Relaying disallowed as Sender@zoho.MyDomain.com

So I all my test passed, the same as I repated them after I entered the Dicourse docker container, as per How do I get into a Docker container's shell? - Stack Overflow

After all my test I suggest that the production.log should also log the answers from the smtp servers as that makes life much easier. Some of my troubles would have been solved much quicker.

It quite confusion if sent or failed emails are logged with the same message. In all my logs I have
Delivered mail 6c663176-73a2-4feb-83f9-32b6113cec27@forum.fairbnb.community (308.9ms)
sometimes it is followed by
Job exception: end of file reached
although Online SMTP tool - Test your Mail Server on the fly tells me differently, using the same settings.
using the email defined in
```
## If you want to set the 'From' email address for your first registration, uncomment and change:
## After getting the first signup email, re-comment the line. It only needs to run once.
- exec: rails r "SiteSetting.notification_email='Discourse@zoho.MyDomain.com'"
```
what is not listed after
```
./launcher rebuild app
 tail shared/standalone/log/rails/production.log
```
but at least the emails failed to be delivered in the first registration process (I did it twice for whatever reason) were still queued and thus send after the change in the app.yml.
After I received the emails from Discourse@zoho.MyDomain.com I did more shared/standalone/log/rails/production.log and found

production.log extracts, newest at the bottom

Delivered mail 63ac46aa-3faf-486d-b51c-92668f7a06fe@forum.fairbnb.community (717.5ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.4ms | Allocations: 141)
Delivered mail 19cd2ae0-6aba-4f3f-9726-883102cb1dc0@forum.fairbnb.community (462.7ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 139)
Delivered mail 0319f11e-9748-4167-bb3e-3aea2e0ac1a5@forum.fairbnb.community (283.9ms)
Job exception: end of file reached
...
  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 139)
Delivered mail 0319f11e-9748-4167-bb3e-3aea2e0ac1a5@forum.fairbnb.community (283.9ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 139)
Delivered mail 6c663176-73a2-4feb-83f9-32b6113cec27@forum.fairbnb.community (308.9ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.5ms | Allocations: 138)
Delivered mail 3315fa23-8f66-4a44-a39d-f367ce1409ba@forum.fairbnb.community (360.0ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 12.3ms | Allocations: 141)
Delivered mail 79e7b7de-f92f-4e39-bd01-f428044a0da3@forum.fairbnb.community (383.9ms)
....
  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 139)
Delivered mail 0319f11e-9748-4167-bb3e-3aea2e0ac1a5@forum.fairbnb.community (283.9ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.3ms | Allocations: 139)
Delivered mail 6c663176-73a2-4feb-83f9-32b6113cec27@forum.fairbnb.community (308.9ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 0.5ms | Allocations: 138)
Delivered mail 3315fa23-8f66-4a44-a39d-f367ce1409ba@forum.fairbnb.community (360.0ms)
Job exception: end of file reached

  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 12.3ms | Allocations: 141)
Delivered mail 79e7b7de-f92f-4e39-bd01-f428044a0da3@forum.fairbnb.community (383.9ms)
Job exception: end of file reached
...
Started POST "/admin/email/test" for 185.45.238.16 at 2019-12-21 16:30:14 +0000
Processing by Admin::EmailController#test as */*
  Parameters: {"email_address"=>"Stefan.Mueller.83@gmail.com"}
  Rendering layouts/email_template.html.erb
  Rendered layouts/email_template.html.erb (Duration: 2.3ms | Allocations: 336)
Delivered mail ef530971-de4e-4460-9e2d-b5a64f88b8b6@forum.fairbnb.community (1169.4ms)
...
Delivered mail 48f1381d-b5e4-4095-8aef-d381001f8a54@forum.fairbnb.community (1143.5ms)
Delivered mail a343fac0-5af3-407b-addc-606a133ff003@forum.fairbnb.community (1216.2ms)
Delivered mail 3713349b-203e-4454-b32b-17522997e8c1@forum.fairbnb.community (1223.0ms)
Delivered mail bce49105-f8e4-4a7a-ab23-db91c1b416ba@forum.fairbnb.community (1268.5ms)

PackElend · December 22, 2019, 12:34am

is not advised see ruby - SSL mode flags - verification of certificates: is it safe to use :none? - Stack Overflow
May you play around with OpenSSL command?

csmu · December 23, 2019, 8:14am

2019-12-23T08:14:13Z
This was blocked until I made the device have a fully qualified domain name (FQDN)

Queth · January 5, 2020, 9:59am

Edit: issues is solved, turned out discourse doctor gave an error while everything was fine.

hello, I hope someone can help me because I’m breaking my head over this and can’t figure out what is wrong.

I have setup the forum successfully, however it doesn’t send mails. For example, mail to reset password. I tried a test mail from the mail test in admin CP and it didn’t arrive.

error: There was a problem sending the test email. Please double-check your mail settings, verify that your host is not blocking mail connections, and try again.

I did all the steps in the troubleshooting first.

i checked the logs and email is getting sent but refused:

Connection refused - connect(2) for “[domainname]" port 587

I have setup the email account at discourse@[domainname], also added a noreply@[domainname].

here are the settings in app.yml.

DISCOURSE_SMTP_ADDRESS: [domainname]
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: discourse@[domainname]
DISCOURSE_SMTP_PASSWORD: [password]
DISCOURSE_SMTP_ENABLE_START_TLS: true
DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: none

what do I need to change/where?

thank you so very much in advance!

minchen · January 4, 2020, 12:43pm

  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: smtp.zoho.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: hi@yourdomain.com
  DISCOURSE_SMTP_PASSWORD: "password8888"
  DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)
  DISCOURSE_SMTP_AUTHENTICATION: login
  DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: none

##- exec: rails r "SiteSetting.notification_email... change to - exec: rails r "SiteSetting.notification_email... (remove the ##)

run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  - exec: rails r "SiteSetting.notification_email='hi@yourdomain.com'"
  - exec: echo "End of custom commands"

I solved the email problem through the above method, I hope this method can help other people.

PackElend · January 4, 2020, 9:01pm

so registration email was send as shown in the guide discourse/INSTALL-cloud.md at master · discourse/discourse · GitHub?
So the setup should be correct.

Have you tried to send an email doing telnet/openssl as described in Troubleshooting email on a new Discourse install

Queth · January 5, 2020, 9:58am

Thank you for your reply! The issue has been solved (sorry I forgot to post here, it’s in another topic) it turned out everything worked ok, but bc I used discourse doctor from the shell to check the settings (I was changing app.yml every time) - it turned out discourse doctor gave a sign something was wrong with mail while in fact it worked…

PackElend · January 6, 2020, 1:38pm

you are welcome. I’m dropping by here frequently; quite often as a last resort .
Quick answers keeping me motivated.

where? I could be helpful in future cases.

Queth · January 8, 2020, 9:37am

Fair enough, I was too lazy to look it up when I posted.

Here it is: Discourse-doctor Error 503 AUTH command used when not advertised - but email works fine from admin CP