A few days ago I upgraded my Discourse from some version I don’t know, however I strongly believe (but can’t swear on it) it was a 2.4 beta, to currently 2.4.0.beta4.
Recently I noticed that e-mail sending is no longer working, I have a lot of failed jobs in Sidekiq. The error for all those jobs is: “Jobs::HandledExceptionWrapper: Wrapped OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: unsupported protocol”.
My e-mail settings point to an old mail server of mine (which otherwise works fine for all clients, of different kinds), port 587 with plain authentication and enable_starttls_auto set to true. It has worked fine since I set up Discourse earlier this year, so I’m pretty sure that it’s since the latest upgrades that it stopped working. The operating system has not changed/been upgraded during this time, nor has the mail server.
I have read Discourse Version 2.4 and am not seeing anything mail or OpenSSL related there.
Q1: Where can I tell from which version the last upgrade and the one before that was done, so I can track the versions I’ve been using?
Q2: Where can I find more specific timestamps for when the mail jobs started failing? I clicked a job in Sidekiq and it tells me it was created two days ago, which I think is in line with when I did the upgrade. But I’d like to verify that mail jobs didn’t fail before that.
Q3: Presumably something relating to OpenSSL changed in the version I started running (relative to the one I was running before). What could this have been, and is there a setting anywhere that I can tweak? Or should I try to downgrade? Or what is there any way that I can get additional from the job processing so I can see what protocol it’s complaining about?
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp
CONNECTED(00000003)
139861698753664:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 353 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_1
CONNECTED(00000003)
140427988595840:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 174 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.1
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1568985038
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_2
CONNECTED(00000003)
140184139936896:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 258 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1568985044
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_3
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 262 bytes and written 278 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
root@foo-app:/#
I’m am seeing an error message “protocol unsupported” when forcing TLS other than 1.3 (using additional arguments to the openssl command, see below), and “write:errno=0” only when forcing TLS 1.3. With your command (not forcing TLS) I’m not seeing any specific version being used/attempted.
If someone could answer questions #1 and #2 it would be helpful. I need to consider if there’s a way to downgrade this ASAP, and in order to do that I need to know what the last working version was.
Please edit /var/discourse/launcher and replace the base image version (image="discourse/base:2.0.20190906-0522") in line 91 with image="discourse/base:2.0.20190625-0946"
Rebuild the container afterwards and run the commands you ran in E-mail sending not working after upgrade. TLS 1.3 won’t work, but is the output of the other commands similar? If not, what’s different?
Could you PM me the hostname of the SMTP in case it is publicly available?
Thank you @gerhard! Your suggestion solved the problem. I changed the base image version, rebuild, and straight away the forum started sending the queued e-mails (about 10k :D).
I did run the commands in the container again and am getting other (successful) output. This output contains certificates and a bunch of other stuff, so unless you really need it I’d prefer to not paste it in here. Let me know if this is a problem and you really need it.
I will PM you the hostname of the mail server so you can debug this issue in more detail - please keep it to yourself Thanks!
Edit: I’m marking your last post as a solution because it solved the problem. However obviously we need to identify what’s causing it in the newer base image, so one can update in the future.
As a workaround, you should be able to add sed commands to the run section at the end of app.yml to remove the following two settings from the /etc/ssl/openssl.cnf file.
Thanks!