E-mail sending not working after upgrade

A few days ago I upgraded my Discourse from some version I don’t know, however I strongly believe (but can’t swear on it) it was a 2.4 beta, to currently 2.4.0.beta4.

Recently I noticed that e-mail sending is no longer working, I have a lot of failed jobs in Sidekiq. The error for all those jobs is: “Jobs::HandledExceptionWrapper: Wrapped OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: unsupported protocol”.

My e-mail settings point to an old mail server of mine (which otherwise works fine for all clients, of different kinds), port 587 with plain authentication and enable_starttls_auto set to true. It has worked fine since I set up Discourse earlier this year, so I’m pretty sure that it’s since the latest upgrades that it stopped working. The operating system has not changed/been upgraded during this time, nor has the mail server.

I have read Discourse Version 2.4 and am not seeing anything mail or OpenSSL related there.

Q1: Where can I tell from which version the last upgrade and the one before that was done, so I can track the versions I’ve been using?

Q2: Where can I find more specific timestamps for when the mail jobs started failing? I clicked a job in Sidekiq and it tells me it was created two days ago, which I think is in line with when I did the upgrade. But I’d like to verify that mail jobs didn’t fail before that.

Q3: Presumably something relating to OpenSSL changed in the version I started running (relative to the one I was running before). What could this have been, and is there a setting anywhere that I can tweak? Or should I try to downgrade? Or what is there any way that I can get additional from the job processing so I can see what protocol it’s complaining about?

One for you @gerhard :wink:

Can you try to connect to the SMTP server from within the Docker container?

openssl s_client -connect <hostname>:<port> -starttls smtp

Does it work? Which protocol does it select? You should see something like this near the end of the output:

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256

or when it’s using TLS 1.3

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384

Here are four runs within Docker:

root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp
CONNECTED(00000003)
139861698753664:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 353 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_1
CONNECTED(00000003)
140427988595840:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 174 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1568985038
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_2
CONNECTED(00000003)
140184139936896:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1922:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 320 bytes and written 258 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1568985044
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
root@foo-app:/#
root@foo-app:/#
root@foo-app:/# openssl s_client -connect mail.foo.com:587 -starttls smtp -tls1_3
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 262 bytes and written 278 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
root@foo-app:/#

I’m am seeing an error message “protocol unsupported” when forcing TLS other than 1.3 (using additional arguments to the openssl command, see below), and “write:errno=0” only when forcing TLS 1.3. With your command (not forcing TLS) I’m not seeing any specific version being used/attempted.

If someone could answer questions #1 and #2 it would be helpful. I need to consider if there’s a way to downgrade this ASAP, and in order to do that I need to know what the last working version was.

현재 내 큐에 실패한 이메일 작업이 3,682개나 쌓여 있습니다. 계정 생성 확인 이메일은 물론, 다른 어떤 이메일도 발송되지 않고 있습니다.

Please edit /var/discourse/launcher and replace the base image version (image="discourse/base:2.0.20190906-0522") in line 91 with image="discourse/base:2.0.20190625-0946"

Rebuild the container afterwards and run the commands you ran in E-mail sending not working after upgrade - #4 by rawtaz. TLS 1.3 won’t work, but is the output of the other commands similar? If not, what’s different?

Could you PM me the hostname of the SMTP in case it is publicly available?

감사합니다, @gerhard! 제안해 주신 방법으로 문제가 해결되었습니다. 베이스 이미지 버전을 변경하고 다시 빌드하자마자 포럼이 대기 중인 이메일(약 10,000통 :D)을 보내기 시작했습니다.

컨테이너에서 명령어를 다시 실행해 보니 다른(성공적인) 출력이 나옵니다. 이 출력에는 인증서와 기타 여러 정보가 포함되어 있으므로, 정말로 필요하지 않으시다면 여기에 붙여넣는 것을 피하고 싶습니다. 이것이 문제가 되거나 정말로 필요하신 경우 알려주세요.

메일 서버의 호스트네임을 개인 메시지로 보내 드리겠습니다. 이렇게 하면 해당 문제를 더 자세히 디버깅하실 수 있을 것입니다. 비밀로 해 주시면 감사하겠습니다 :slight_smile: 감사합니다!

수정: 문제를 해결해 주셨으므로 마지막 게시글을 해결책으로 표시하겠습니다. 다만, 향후 업데이트를 위해 새로운 베이스 이미지에서 이 문제를 일으키는 원인을 분명히 파악할 필요가 있습니다.

네, 기존 이미지를 계속 사용하는 것은 장기적인 해결책이 아닙니다. 해당 이미지는 사실상 더 이상 지원되지 않으므로, 나중에 문제가 생길 수 있습니다…

이 문제는 Email SSL Errors after Update to 2.4.0.beta4 와 유사합니다. SMTP를 TLSv1.2를 사용하고 DH 키 크기를 최소 2048비트로 설정하도록 업데이트하는 것을 권장합니다.

임시 해결책으로 app.yml 끝부분의 run 섹션에 sed 명령을 추가하여 /etc/ssl/openssl.cnf 파일에서 다음 두 설정을 제거할 수 있습니다.

MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2