There are a lot of posts about this in #support so now we seem to have found the cause I thought it might be good to create a #bug topic about this containing only the actual issue described here Email Hostname Certificate Mismatch Causing sidekiq Queue Overload, Severe Site Instability - #47 by RGJ
Long story short:
Since 2.9.0 beta 4 the settings
DISCOURSE_SMTP_OPENSSL_VERIFY_MODE and DISCOURSE_SMTP_ENABLE_START_TLS do not work correctly any longer.
This seems to be related to the move to Rails 7, which updated net-smtp from 0.1.0 to 0.3.1, which changed the defaults for these two connection parameters.
The way the smtp gem calls net-smtp does not disable enable_starttls_auto and openssl_verify_mode, it only enables it when enabled.
Related report for the smtp gem: SMTP: allow disabling starttls_auto since it's now true by default in Ruby 3 by jeremy · Pull Request #1435 · mikel/mail · GitHub
Technically this bug is outside the Discourse code since this is happening in the smtp gem, but since that is currently forked for Discourse I think this does warrant a #bug topic. If it cannot be fixed easily then these parameters could be removed from the docs and maybe a deprecation warning could be shown so at least all the confusion is gone.