There have been a lot of updates recently. One busted the subject line (no longer includes the category) and now no mail goes out at all. Users are very unhappy. I don’t know how to start debugging this issue. Where can I error logs and I recall there is some page about sidekiq queues and so on, but I cant find it. Any help would be most appreciated.
Yeah, I’ve noticed that email notifications don’t seem to be firing at the moment after an update yesterday, although digest/summaries still are. Are we alone in this?
2 likes
The cause of this may be sidekiq failing to process scheduled jobs when it should.
We identified the same problem earlier today on our CD sites. Make sure you are at least the commit:
(I think this is the commit, not 100% sure)
To see if the problem is the same, check scheduled jobs in /sidekiq and see if there’s any in the past.
3 likes
Yup, we were caught up in that. An update has sorted it.
2 likes
4 posts were split to a new topic: Email From: headers lost their “via SITENAME” text
I confirm hundreds of failed sidekiq jobs on latest-release +103
fixed on latest-release +153
I am updated to latest and am still having an email sending problem on one of my sites. I just get an error message when sending a test email.
ERROR - end of file reached
On mobile now we’ll check sidekiq and logs when I am at my computer. Any other suggestions where to look?
Backtrace
Message (5 copies reported)
Job exception: end of file reached
Backtrace
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:237:in `rbuf_fill'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:199:in `readuntil'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:209:in `readline'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:1017:in `recv_response'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:676:in `block in do_start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:1027:in `critical'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:676:in `do_start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:642:in `start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:154:in `start_smtp_session'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:108:in `deliver!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/message.rb:269:in `deliver!'
/usr/local/lib/ruby/3.3.0/delegate.rb:87:in `method_missing'
/var/www/discourse/lib/email/sender.rb:296:in `send'
/var/www/discourse/app/jobs/regular/download_backup_email.rb:19:in `execute'
/var/www/discourse/app/jobs/base.rb:318:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/app/jobs/base.rb:305:in `block in perform'
/var/www/discourse/app/jobs/base.rb:301:in `each'
/var/www/discourse/app/jobs/base.rb:301:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/suppress_user_email_errors.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/discourse_event.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/pausable.rb:131:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in `local'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/config.rb:39:in `block in <class:Config>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in `global'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:10:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:19:in `block in safe_thread'
Just tried the email sending test again on my laptop and get this:
There was a problem sending the test email. Please double-check your mail settings, verify that your host is not blocking mail connections, and try again.
And in the javascript console:
Maybe my problem is different - I will check with mailgun.
Edit: I did see two errors in the retries and deleted them. Still getting the same error now when testing, unfortunately. I also checked the smtp server and that appears to be working.
Hi Tobias!
Your problem is different - the connection is getting hung up waiting for a response shortly after initial successful connection.
I’d hazard a guess that you’re trying to speak the wrong protocol to the wrong port… what settings are you using?
Does the rake emails:test task (with recently updated logic and error messages) show any different error?
Hi Michael! Thanks for the response. I miss you guys so much! 
Hmm.. I just moved my site from DO to Hetzner and it worked fine for a couple of weeks. My other site is working fine too. It’s a puzzle. Only sometime about a week ago it stopped working and when I looked into it saw the errors. I reached out to Hetzner (declined to help) and to Mailgun. According to Mailgun:
Thank you for your reply, the last accepted authenticated event we are seeing was on Jan 11th and was sent via SMTP.
Can you please confirm if any changes have been made? Please provide a screenshot of your sending application configuration for our review, as well as any relevant errors on your sending application/ SMTP sending logs.
I just changed my mailgun password in case that was it and tried again, but no dice.
rake emails:test output:
root@ubuntu-4gb-nbg1-1-app:/var/www/discourse# rake emails:test
Testing sending to using smtp.mailgun.org:587, username:postmaster@domain with plain auth.
====================================================================================== ERROR =======================================================================================
UNKNOWN ERROR!
EOFError: end of file reached
===================================================================================== SOLUTION =====================================================================================
This is not a common error. No recommended solution exists!
Please report the exact error message above to https://meta.discourse.org/
(And a solution, if you find one!)
====================================================================================================================================================================================
1 like
I think it’s failing before even attempting login.
To eliminate Discourse as a factor, try from the host AND from inside the container:
$ openssl s_client -connect smtp.mailgun.org:587 -starttls smtp
You should get a bunch of output and then be able to try to auth:
○ → openssl s_client -connect smtp.mailgun.org:587 -starttls smtp
Connecting to 34.160.63.108
CONNECTED(00000003)
…
SSL-Session:
…
---
read R BLOCK
EHLO localhost
250-2ed1d46f4d7dec773e2a97b59f3a3bf8a2d6db54f94eead5dcf49e3ea1caac18
250-AUTH PLAIN LOGIN
250-SIZE 52428800
250-8BITMIME
250-SMTPUTF8
250 PIPELINING
AUTH PLAIN bWljaGFlbABtaWNoYWVsAHBhc3N3b3Jk
501 Username used for auth is not valid email address
535 Authentication failed
closed
The strings you would type are:
EHLO localhost
AUTH PLAIN bWljaGFlbABtaWNoYWVsAHBhc3N3b3Jk
(that string is the creds michael/password so it obviously won’t work, but you can see this post to learn how to construct the string for your actual credentials if you want to try by hand)
Hopefully seeing firsthand what works and what fails will help.
You might also want to try using swaks if it’s available - it’s probably an OS package you can install.
It’s a bit easier and you can e.g.:
swaks --to frodo@shire.net --from bilbo@shire.net --auth PLAIN --auth-user bilbo --auth-password ring --server smtp.mailgun.org:587 --tls
except you can use your real credentials.
The output of that might also help reveal the problem.
I tried swaks and got this:
=== Trying smtp.mailgun.org:587...
=== Connected to smtp.mailgun.org.
*** Remote host closed connection unexpectedly.
That inspired me to check from my other server, where swaks had “Great success” - the message is quite adorable!
<~ 250 Great success
~> QUIT
<~ 221 See you later. Yours truly, Mailgun
=== Connection closed with remote host.
So the issue is either with mailgun blocking my server, or my server somehow being misconfigured. I’ll check with mailgun and then if that’s not it then I’ll destroy and rebuild my server.
1 like
Makes sense; this is essentially the same error as
As you suspect, the most likely cause is something external interfering with the connection.
1 like
i have a suspicion you need to use port 2525 instead of 587
Hetzner explicitly states they do not block port 587.
Also, if they did it would probably show up as a failure to establish a connection.
That almost rules out Discourse and Mailgun configuration.
At this point the most useful diagnostic is to try an alternate Mailgun submission port from the affected server:
openssl s_client -connect smtp.mailgun.org:2525 -starttls smtp
(or the same test with swaks).
Mailgun supports 2525 specifically for environments where 587 is interfered with by firewalls, egress filtering, or provider-level network rules.
If:
- 2525 works and 587 doesn’t → very likely network / IP / routing interference
- both fail → even stronger case that something external is blocking or terminating the connection
Either way, the behaviour matches “external connection interference” much more closely than a Discourse or Sidekiq regression.
I haven’t heard back from mailgun yet.. my plan if they can’t help is to just start over with a new hetzner server.
I did try the swaks with other ports, and interestingly I got the same error with 2525, and immediately without any delay.
=== Trying smtp.mailgun.org:2525...
=== Connected to smtp.mailgun.org.
*** Remote host closed connection unexpectedly.
But got a different error with 25 and 465, and it took some seconds for the response to come.
=== Trying smtp.mailgun.org:25...
*** Error connecting to smtp.mailgun.org:25:
*** Connection timed out
=== Trying smtp.mailgun.org:465...
*** Error connecting to smtp.mailgun.org:465:
*** Connection timed out
Not sure what to make of that. Maybe there is a misconfigured firewall on the server blocking some ports.
that is intentional and expected.
Hetzner blocks those ports by default.
1 like
Makes sense.. but it’s interesting that 25 and 465 took longer to fail than the other ports. Anyway, this is not urgent and I will wait to hear back from Mailgun. This is for my family site so I am just encouraging everyone to log in and check for notifications for a change, and not just wait for email!
Heading to Germany for a visit to family and a funeral tomorrow.. next week I’ll look at this again and likely just start over with a new server.
Thanks for all the guidance! I learned alot through this process. Really like that swaks tool.
Timing can be useful information indeed.
When a port (more generally, traffic) is blocked, it can be blocked by dropping (silent rejection) or rejected (an “unreachable” message (e.g. port unreachable) is sent back to the originator.
A silent drop causes when you see with 25/465 - a connection attempt is made and nothing happens until the timeout is reached.
A rejection causes a message such as “Connection refused” or “Port unreachable”.
That’s indicative of intended behaviour - you’re getting a connection and then the connection is being immediately closed.
You’ve varied the source, so something else you can try is varying the destination. Try the same command against e.g. smtp.gmail.com or smtp.office365.com. You should get an authentication failure, and that’s a strong indication it’s mailgun rejecting you specifically.
1 like
That explanation from @supermathie is spot-on 
The timing difference is actually useful signal, not noise.
To summarise what you’re seeing:
- 25 / 465 timing out → silent drop (Hetzner policy-level block, expected)
- 2525 connecting then immediately closing → TCP path is fine, but the remote side is terminating the session
That last one is the key detail.
Because:
- the TCP handshake succeeds
- TLS negotiation never really begins
- and it happens instantly
…it strongly suggests Mailgun is rejecting the connection early, not Discourse, Sidekiq, or Ruby.
This lines up with a few things we’ve seen recently:
- IP reputation / region-based filtering
- new Hetzner IP ranges not yet trusted
- or Mailgun-side policy checks before SMTP banner exchange
Nothing in Discourse would cause immediate remote socket closure like that - Sidekiq just hands the message off to Net::SMTP and waits.
If you want one more very strong confirmation later (no urgency at all):
openssl s_client -connect smtp.gmail.com:587 -starttls smtp
You should get a proper SMTP banner and then an auth failure - which would basically prove outbound SMTP itself is fine and Mailgun is the only endpoint behaving differently.
And completely understand pausing this - especially given the circumstances.
I’m really sorry you’re dealing with that on top of everything else 
If Mailgun comes back with something useful, great - but honestly, starting fresh or switching providers (Brevo, Postmark, etc.) is often faster than waiting on SMTP reputation fixes.
You’ve already debugged this exactly the right way - nothing here suggests you missed something or misconfigured Discourse.
Safe travels to Germany, and take care.
1 like