Email SSL Errors after Update to 2.4.0.beta4

Installation
1

Since upgrading to 2.4.0.beta4, none of my installs that are using Rackspace for outgoing email have been able to send, well, outgoing email. Given that the outgoing email server is, again, Rackspace, I am going to presume that their SSL/TLS settings are correct (and, at any rate, they appear to work in every major email client). Perhaps this thread is related.

Although, after applying recent updates (I am unsure as to precisely which change), the error is no longer the same as that mentioned in the aforementioned thread, but, instead, is now this:

Jobs::HandledExceptionWrapper: Wrapped Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)>

I presume this is a bug.

Edit: Another related thread

@Gerhard

1 Like
2

Can you try to connect to the SMTP server from within the Docker container?

SMTP with StartTLS (default, unless you changed DISCOURSE_SMTP_ENABLE_START_TLS in app.yml):

openssl s_client -connect <hostname>:<port> -starttls smtp

SMTP

openssl s_client -connect <hostname>:<port>
3 Likes
3

With the -starttls flag, it simply returns “CONNECTED”. Without -starttls:

root@omnifora-com-app:/var/www/discourse# openssl s_client -connect secure.emailsrvr.com:465
CONNECTED(00000003)
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL, CN = secure.emailsrvr.com
verify return:1
139636332590208:error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small:../ssl/statem/statem_clnt.c:2156:
---
Certificate chain
 0 s:OU = Domain Control Validated, OU = EssentialSSL, CN = secure.emailsrvr.com
   i:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
 1 s:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
   i:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
 2 s:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
   i:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
 3 s:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
   i:C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIG5jCCBc6gAwIBAgIRAMWoQ0lmf1VC8Ch8zZZTHm0wDQYJKoZIhvcNAQELBQAw
gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD
VQQDEy1DT01PRE8gUlNBIERvbWFpbiBWYWxpZGF0aW9uIFNlY3VyZSBTZXJ2ZXIg
Q0EwHhcNMTkwMTEwMDAwMDAwWhcNMjAwMzEwMjM1OTU5WjBZMSEwHwYDVQQLExhE
b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxFTATBgNVBAsTDEVzc2VudGlhbFNTTDEd
MBsGA1UEAxMUc2VjdXJlLmVtYWlsc3J2ci5jb20wggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDZzFpkI/ujPCuNZpHLueu+/iqUsc5U7+yYa9d6xIbkh2BN
u+OpBNCTn4ACa0a3EaqRVyceUUh8TodUPtkZYLZO6iqwl2eOd8h3NXxtRlyaj0Hz
uSOlRbA5CiVZ4H1Ia8k/DVh+r1Rk6Da/f52wBJE8ICFgm7Uyrjtfcc90gBk+7i4I
y1aNwKW/nqmqQBEiTeyUF2kJiTovtorQo7zaedPefm2VUoKyxe/8jl7qA7F9+1p0
XvvWrc3/vqEEZR6tmcAF8tmp0MSkMnt3klwg/xopVn5nPq52t6fLRXA0aLFBUHzT
U82Iw1Weg+gUVi77ONDIabfYuCqqEgpnAyeUhh8hAgMBAAGjggNvMIIDazAfBgNV
HSMEGDAWgBSQr2o6lFoL2JDqElZz30O0Oija5zAdBgNVHQ4EFgQUFJzBKVTbToPC
UoxWxXfRAJGz+YswDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0l
BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCME8GA1UdIARIMEYwOgYLKwYBBAGyMQEC
AgcwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLmNvbS9DUFMw
CAYGZ4EMAQIBMFQGA1UdHwRNMEswSaBHoEWGQ2h0dHA6Ly9jcmwuY29tb2RvY2Eu
Y29tL0NPTU9ET1JTQURvbWFpblZhbGlkYXRpb25TZWN1cmVTZXJ2ZXJDQS5jcmww
gYUGCCsGAQUFBwEBBHkwdzBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5jb21vZG9j
YS5jb20vQ09NT0RPUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNy
dDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMDkGA1UdEQQy
MDCCFHNlY3VyZS5lbWFpbHNydnIuY29tghh3d3cuc2VjdXJlLmVtYWlsc3J2ci5j
b20wggGABgorBgEEAdZ5AgQCBIIBcASCAWwBagB3ALvZ37wfinG1k5Qjl6qSe0c4
V5UKq1LoGpCWZDaOHtGFAAABaDf6Nt0AAAQDAEgwRgIhAJDqOzt2LWqviVrjKGFL
UCPuu/HWeuILG/7VuDwJWWYYAiEAvCaXH3lSCRWOgGquaz9lW3uITCuKQP0TOOMv
JPbcN/IAdwBep3P531bA57U2SH3QSeAyepGaDIShEhKEGHWWgXFFWAAAAWg3+jcn
AAAEAwBIMEYCIQCxU8IX94IoSwsrpo6zJoUMO4uNGuTkpLSY0h/KWbspqQIhAIy4
XfY5RtTTLpB3EFLXMyQSL9/gyNpfJ1OtbYtOkL0pAHYA8JWkWfIA0YJAEC0vk4iO
rUv+HUfjmeHQNKawqKqOsnMAAAFoN/o5AAAABAMARzBFAiAePxbn6JuVUkYjBVnF
MPHeqyqAaYpdwyGxaC3Cz4WZhAIhAPFXU3e0+7GkNMjXFPQ6UMd55zeUJcxakFIt
ggm7ioLYMA0GCSqGSIb3DQEBCwUAA4IBAQAkLuNWuHt5GOXkzJlys09mg22+MnhF
4y+abm7F54stsv0A2Gc4my4bEXOZ4ozf0g1Yjb/ZVlSVrNC125CSnXd6bEcesjcn
c3oxO+9dFCQGMH4CZPVSoDKBk41+VP9IcnfibhSzV8wFXQh+Tt1OpRoNgqM888Es
JvYP9B2OgDvQFnDNAcJXM5fgX1CilyXqPtz2QYDNVgN8tuRSRPlaGTkZgGMsCO12
GjxLD5UGsxh5c08KSRgd4Uv6BRH/hE62spqvmDUDzuU+Qx9N4/Tz2ocv8LI8GlqV
RYOe+6lLe8t33yH0dnRWKGrpT8gWkul1qLHI9I7LYZMvMKdcxl8oBBGF
-----END CERTIFICATE-----
subject=OU = Domain Control Validated, OU = EssentialSSL, CN = secure.emailsrvr.com

issuer=C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA

---
No client certificate CA names sent
---
SSL handshake has read 6414 bytes and written 319 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1569003408
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
1 Like
4

It would appear the issue has returned to:

Jobs::HandledExceptionWrapper: Wrapped OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: dh key too small

(At least for the past few thousand failures.)

Also: Happy birthday, @gerhard.

2 Likes
5

That error suggests that the SMTP server is configured badly and uses a DH key which OpenSSL considers as too small.

3 Likes
6

That seems somewhat unlikely from Rackspace, but I can try to rope one of their techs into this.

1 Like
7

Well, the error comes from OpenSSL and AFAIK we are using the defaults provided by Debian / Ruby? :man_shrugging:

4 Likes
8

It seems like the new version of the base image packages an OpenSSL version that finally deprecated old insecure signing algorithms.

Our company intranet has an old windows CA that used md5 and it completely broke HTTPS on my Discourse installation after upgrading. Nginx complained about “SSL_CTX_use_certificate:ca md too weak” and refused to load the https cert.

RHEL and Centos has a legacy mechanism to turn them back on but I couldn’t find a similar compatibility setting in Debian/Ubuntu
https://bugzilla.redhat.com/show_bug.cgi?id=1335914
https://bugzilla.redhat.com/show_bug.cgi?id=1498322
https://github.com/kubernetes/ingress-nginx/issues/3571

I’m sure more people are going to run into this with the amount of old, insecure certificates floating around but there’s probably not much to do about it other than to get the certificates replaced. I would try contacting Rackspace directly about the email issue.

3 Likes
9

This is a question for @gerhard

2 Likes
10

Pretty sure this is a bug. I had a Rackspace technician look into it and he provided the following information about their DH key:

Here is the publicly available information:

CA = Comodo Limited CA
Certificate Key Size = 2048 bit
Domain Name = mx1.emailsrvr.com and mx2.emailsrvr.com
Email server hostname = secure.emailsrvr.com
Mail Host Software (Identify the software and version is running on the MTA) =
ecelerity 2.2.3.49
Cipher Strength = AES256-SHA

I should think a 2048-bit key would not (accurately) be considered “too small”.

1 Like
11

From U.S. | Let There Be Change | Accenture

The currently recommended minimum size for DH parameters is 2048 bits. Anything equal or below 1024 is considered insecure.

Okay, so let’s take a look at the DH key by using an older version of Debian:

docker run --rm -it debian:stretch
apt update && apt install -y openssl
openssl s_client -connect secure.emailsrvr.com:465 | grep "Server Temp Key"

Yeah, the DH key is definitely too small:

Server Temp Key: DH, 1024 bits

I’d say that’s something for Rackspace to fix. As a workaround, you should be able to edit /etc/ssl/openssl.cnf and remove the CipherString = DEFAULT@SECLEVEL=2 at the end of the file. Sidekiq should pick up the new OpenSSL settings after restarting the container.

5 Likes
12 
openssl s_client -connect secure.emailsrvr.com:465 | grep "Server Temp Key"
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL, CN = secure.emailsrvr.com
verify return:1
Server Temp Key: DH, 1024 bits

Back to chatting with Rackspace.

1 Like
16

Update from Rackspace:

Thank you for your patience, and thank you for bringing this to our intention. We can confirm that our current DH Key is 1024 bits. Our Product and Engineering teams have acknowledged that this needs to be increased, and have plans in place to fix this.

I do not have an exact date for you when the fix will be rolled, though the goal is sometime this month. As soon as we do increase the DH Key size we will be sure to provide you with an update.

Thanks again for bringing this to our attention! If you have any additional questions or concerns, please let us know!

I’ll update this thread when I receive notification that the DH key has been upgraded.

4 Likes
17

Was able to fix my problem with that patch, but it´s not very promissing :slight_smile:
On next rebuild it will be gone, again, right ? :slight_smile:

1 Like
18

Well, hopefully Rackspace will fix the issue before you need to do a rebuild. Otherwise you can modify openssl.cnf from within the app.yml by using sed commands to make it a permanent change.

2 Likes
19

Update from Rackspace:

Thank you for your patience. The DH Key size has been updated and now matches the Certificate Key Size. Please test, and let us know if you have any additional questions or concerns!

Verified:

openssl s_client -connect secure.emailsrvr.com:465 | grep "Server Temp Key"
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL, CN = secure.emailsrvr.com
verify return:1
Server Temp Key: DH, 2048 bits

I further verified that I am able to send emails from my Discourse installs again. So (at least for Rackspace) this issue is resolved.

6 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.