There have been a lot of updates recently. One busted the subject line (no longer includes the category) and now no mail goes out at all. Users are very unhappy. I don’t know how to start debugging this issue. Where can I error logs and I recall there is some page about sidekiq queues and so on, but I cant find it. Any help would be most appreciated.
Yeah, I’ve noticed that email notifications don’t seem to be firing at the moment after an update yesterday, although digest/summaries still are. Are we alone in this?
2 Likes
The cause of this may be sidekiq failing to process scheduled jobs when it should.
We identified the same problem earlier today on our CD sites. Make sure you are at least the commit:
(I think this is the commit, not 100% sure)
To see if the problem is the same, check scheduled jobs in /sidekiq and see if there’s any in the past.
2 Likes
Yup, we were caught up in that. An update has sorted it.
2 Likes
4 posts were split to a new topic: Email From: headers lost their “via SITENAME” text
I confirm hundreds of failed sidekiq jobs on latest-release +103
fixed on latest-release +153
I am updated to latest and am still having an email sending problem on one of my sites. I just get an error message when sending a test email.
ERROR - end of file reached
On mobile now we’ll check sidekiq and logs when I am at my computer. Any other suggestions where to look?
Backtrace
Message (5 copies reported)
Job exception: end of file reached
Backtrace
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:237:in `rbuf_fill'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:199:in `readuntil'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:209:in `readline'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:1017:in `recv_response'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:676:in `block in do_start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:1027:in `critical'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:676:in `do_start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/net-smtp-0.5.1/lib/net/smtp.rb:642:in `start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:154:in `start_smtp_session'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/network/delivery_methods/smtp.rb:108:in `deliver!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mail-2.9.0/lib/mail/message.rb:269:in `deliver!'
/usr/local/lib/ruby/3.3.0/delegate.rb:87:in `method_missing'
/var/www/discourse/lib/email/sender.rb:296:in `send'
/var/www/discourse/app/jobs/regular/download_backup_email.rb:19:in `execute'
/var/www/discourse/app/jobs/base.rb:318:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `with_connection'
/var/www/discourse/app/jobs/base.rb:305:in `block in perform'
/var/www/discourse/app/jobs/base.rb:301:in `each'
/var/www/discourse/app/jobs/base.rb:301:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/suppress_user_email_errors.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/discourse_event.rb:6:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/lib/sidekiq/pausable.rb:131:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job/interrupt_handler.rb:9:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in `track'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in `local'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/config.rb:39:in `block in <class:Config>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in `global'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `handle_interrupt'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:10:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:19:in `block in safe_thread'
Just tried the email sending test again on my laptop and get this:
There was a problem sending the test email. Please double-check your mail settings, verify that your host is not blocking mail connections, and try again.
And in the javascript console:
Maybe my problem is different - I will check with mailgun.
Edit: I did see two errors in the retries and deleted them. Still getting the same error now when testing, unfortunately. I also checked the smtp server and that appears to be working.
Hi Tobias!
Your problem is different - the connection is getting hung up waiting for a response shortly after initial successful connection.
I’d hazard a guess that you’re trying to speak the wrong protocol to the wrong port… what settings are you using?
Does the rake emails:test task (with recently updated logic and error messages) show any different error?
Hi Michael! Thanks for the response. I miss you guys so much! 
Hmm.. I just moved my site from DO to Hetzner and it worked fine for a couple of weeks. My other site is working fine too. It’s a puzzle. Only sometime about a week ago it stopped working and when I looked into it saw the errors. I reached out to Hetzner (declined to help) and to Mailgun. According to Mailgun:
Thank you for your reply, the last accepted authenticated event we are seeing was on Jan 11th and was sent via SMTP.
Can you please confirm if any changes have been made? Please provide a screenshot of your sending application configuration for our review, as well as any relevant errors on your sending application/ SMTP sending logs.
I just changed my mailgun password in case that was it and tried again, but no dice.
rake emails:test output:
root@ubuntu-4gb-nbg1-1-app:/var/www/discourse# rake emails:test
Testing sending to using smtp.mailgun.org:587, username:postmaster@domain with plain auth.
====================================================================================== ERROR =======================================================================================
UNKNOWN ERROR!
EOFError: end of file reached
===================================================================================== SOLUTION =====================================================================================
This is not a common error. No recommended solution exists!
Please report the exact error message above to https://meta.discourse.org/
(And a solution, if you find one!)
====================================================================================================================================================================================
I think it’s failing before even attempting login.
To eliminate Discourse as a factor, try from the host AND from inside the container:
$ openssl s_client -connect smtp.mailgun.org:587 -starttls smtp
You should get a bunch of output and then be able to try to auth:
○ → openssl s_client -connect smtp.mailgun.org:587 -starttls smtp
Connecting to 34.160.63.108
CONNECTED(00000003)
…
SSL-Session:
…
---
read R BLOCK
EHLO localhost
250-2ed1d46f4d7dec773e2a97b59f3a3bf8a2d6db54f94eead5dcf49e3ea1caac18
250-AUTH PLAIN LOGIN
250-SIZE 52428800
250-8BITMIME
250-SMTPUTF8
250 PIPELINING
AUTH PLAIN bWljaGFlbABtaWNoYWVsAHBhc3N3b3Jk
501 Username used for auth is not valid email address
535 Authentication failed
closed
The strings you would type are:
EHLO localhost
AUTH PLAIN bWljaGFlbABtaWNoYWVsAHBhc3N3b3Jk
(that string is the creds michael/password so it obviously won’t work, but you can see this post to learn how to construct the string for your actual credentials if you want to try by hand)
Hopefully seeing firsthand what works and what fails will help.
You might also want to try using swaks if it’s available - it’s probably an OS package you can install.
It’s a bit easier and you can e.g.:
swaks --to frodo@shire.net --from bilbo@shire.net --auth PLAIN --auth-user bilbo --auth-password ring --server smtp.mailgun.org:587 --tls
except you can use your real credentials.
The output of that might also help reveal the problem.
I tried swaks and got this:
=== Trying smtp.mailgun.org:587...
=== Connected to smtp.mailgun.org.
*** Remote host closed connection unexpectedly.
That inspired me to check from my other server, where swaks had “Great success” - the message is quite adorable!
<~ 250 Great success
~> QUIT
<~ 221 See you later. Yours truly, Mailgun
=== Connection closed with remote host.
So the issue is either with mailgun blocking my server, or my server somehow being misconfigured. I’ll check with mailgun and then if that’s not it then I’ll destroy and rebuild my server.
1 Like
Makes sense; this is essentially the same error as
As you suspect, the most likely cause is something external interfering with the connection.
1 Like