I decided to spend one more half hour to dig into this and I think I found the cause.
This seems to be related to the move to Rails 7, which updated net-smtp from 0.1.0 to 0.3.1, which changed the defaults.
The way the smtp gem calls net-smtp does not disable enable_starttls_auto and openssl_verify_mode, it only enables it when enabled.