That error suggests that the SMTP server is configured badly and uses a DH key which OpenSSL considers as too small.