Reworking the emails:test rake task output

[supermathie (Michael Brown)]

I recently went through the emails:task and related code with the aim of testing each of the failure codepaths and touching up the error text.

I also discovered that the effects of setting DISCOURSE_SMTP_ENABLE_START_TLS=false are (mostly) nil. Setting this didn’t actually disable STARTTLS and in fact, it can coexist just fine with tls at connect time (DISCOURSE_SMTP_FORCE_TLS=true).

So I’ve:

• removed the code that prevents both from being set: https://github.com/discourse/discourse/pull/35634/commits/0793898fad60871f942c3baa752172268d1360ec
• fixed the code to actually disable STARTTLS when requested: https://github.com/discourse/discourse/pull/35634/commits/e4f8b1f9efdb6f7ef9f5087be14f599ad8746230
• redone the code that looks for failures and updated the text: https://github.com/discourse/discourse/pull/35634/commits/00430529459ef0b189d7576678aeeed32f1a24ee

Before I merge this though, I think an admin warning in the dashboard for when DISCOURSE_SMTP_ENABLE_STARTTLS=false is being set is suitable. I imagine there’s at least one self-hoster out there who has set this but doesn’t need it and is actually relying on STARTTLS.

[pfaffman (Jay Pfaffman)]

That sounds like good work! One thing that I (think I’ve) noticed is that the rake task doesn’t actually use the same code as the actual sending does (like from the /admin/email test page). I’m pretty sure that I’ve had a case where it worked in the UX but not the rake task (or maybe it was the opposite?)

While it’s fresh on your mind, it you could see that at least when it actually sends it does so using the same code that Discourse does, that would be great.

[supermathie (Michael Brown)]

We’re also working on that, and on improving the logging when a queued email job fails.

[Moin]

Is there anything I need to do on the forum you host?

[pmusaraj (Penar Musaraj)]

No, this alert shouldn’t show up on our hosting. We’ll fix it, thanks for the report as always.

[pmusaraj (Penar Musaraj)]

@supermathie

Before I merge this though, I think an admin warning in the dashboard for when DISCOURSE_SMTP_ENABLE_STARTTLS=false is being set is suitable.

@supermathie is this worth PMing every admin on every site that has this variable set? Our current problem checks system will do that, and I’m not sure it is warranted here, given that for the most part, this has a nil effect. ideally, I’d want to only show this on the dashboard without pinging admin users, not sure our current problem checks structure supports that use case.

[supermathie (Michael Brown)]

I think it is - I find it extremely likely that given how confused many admins are around email setup, someone has this variable set when they actually depend on starttls.

Nobody should have it set, probably.

Is rather have a spurious warning than silently break someone’s email setup.

The alternative is removing the check and neutering the variable so it does nothing at all.

[chrisc (Chris Croome)]

It would be nice if the warning didn’t show up when the outgoing SMTP server is the localhost (ie matches the Discourse domain name) as TLS between the Docker container and the host isn’t needed as they are the same machine.

[supermathie (Michael Brown)]

In this case you should remove the variable from the environment.

Discourse will only use STARTTLS if it’s offered.

[josy38 (Johann Christoph Lindner)]

In the current version, I had the case where the commented-out line (default is true) together with force_tls prevented email sending. So I commented it in and set it to false. Email sending now works - but I have this message in the backend…

It depends a bit on which port your mail provider runs on… both don’t work - and definitely have an impact.

[J5lx]

I also noticed this. After updating Discourse on 28 November, I started seeing mail delivery failures with the error message Job exception: :enable_starttls and :tls are mutually exclusive. Set :tls if you're on an SMTPS connection. Set :enable_starttls if you're on an SMTP connection and using STARTTLS for secure TLS upgrade.

Backtrace

/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:159:in `build_smtp_session'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:154:in `start_smtp_session'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:108:in `deliver!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/message.rb:269:in `deliver!'
/usr/local/lib/ruby/3.3.0/delegate.rb:87:in `method_missing'
/var/www/discourse/lib/email/sender.rb:296:in `send'
/var/www/discourse/app/jobs/regular/user_email.rb:80:in `send_user_email'
/var/www/discourse/app/jobs/regular/user_email.rb:40:in `execute'
/var/www/discourse/app/jobs/base.rb:318:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/app/jobs/base.rb:305:in `block in perform'
/var/www/discourse/app/jobs/base.rb:301:in `each'
/var/www/discourse/app/jobs/base.rb:301:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/suppress_user_email_errors.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/discourse_event.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/pausable.rb:131:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in `local'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/config.rb:39:in `block in <class:Config>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in `global'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:10:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:19:in `block in safe_thread'

Just like Johann Christoph, I previously had DISCOURSE_SMTP_ENABLE_START_TLS commented out and DISCOURSE_SMTP_FORCE_TLS set to true for implicit TLS and it worked like a charm. But now I have to explicitly set DISCOURSE_SMTP_ENABLE_START_TLS to false for email delivery to work, which of course triggers the aforementioned message in the admin dashboard.

[supermathie (Michael Brown)]

@josy38

together with force_tls prevented email sending

It will be force_tls preventing email sending - this enforces TLS at connection time and is probably only suitable for port 465.

@J5lx

But now I have to explicitly set DISCOURSE_SMTP_ENABLE_START_TLS to false for email delivery to work

A unicorn!

What SMTP parameters are you using (minus username/password)?

[J5lx]

Port 465 with ENABLE_START_TLS=false, FORCE_TLS=true and OPEN_TIMEOUT=10. That last one is due to strange timeout errors that I started seeing after a system update around three years ago, but I’ll eat my hat if that’s related to the issue described above.

I had a look at the commit history in the meantime and noticed the mail gem was updated to version 2.9.0 just three days before the update that introduced the issue on my end (#36254). There’s an entry in the release notes of that which says “SMTP: refactor and accept starttls :always and :auto by @eval in #1536”. I don’t know much about Ruby, but this change in the referenced PR looks kind of suspicious to me:

Given that DISCOURSE_SMTP_ENABLE_START_TLS is documented to be true by default (i.e. if commented out), is it possible that this is the root of the issue?

[josy38 (Johann Christoph Lindner)]

@J5lx

DISCOURSE_SMTP_FORCE_TLS

As I wrote - I set force_TLS:

DISCOURSE_SMTP_FORCE_TLS = true

and ENABLE_START_TLS explicitly to false… then mail sending works - provided the mail server runs on 465.

With mail servers running on 567 - it must be exactly the other way around.

Since then, I have a notice on the dashboard in the backend… but mail sending works without problems.

[josy38 (Johann Christoph Lindner)]

To be able to set DISCOURSE_SMTP_ENABLE_START_TLS to false, you must comment it out—as it is set to true by default otherwise. That is exactly what caused the problem for me. However, this only happened with the latest build (2025.12.0-latest).

[supermathie (Michael Brown)]

@J5lx

but this change in the referenced PR looks kind of suspicious to me

Wow, this was a poor change.

enable_starttls will REQUIRE the use of starttls, but enable_starttls_auto is opportunistic - it’ll only negotiate tls if it’s offered.

And if the mail server was connected to via initial TLS, it won’t offer starttls:

○ → openssl s_client -connect localhost:5587 -starttls smtp
250 CHUNKING
EHLO localhost
250-testmailrelay
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 CHUNKING

why in the world would they have done that?

The difficulty here is that we should never have offered this configuration in the first place, it should have been something like:

DISCOURSE_SMTP_TLS_MODE = starttls_auto # [ none | starttls | starttls_auto (default) | tls ]

[supermathie (Michael Brown)]

In the interest of trying to have people do less configuration, not more, I think this approach is best:

https://github.com/discourse/discourse/pull/36706