Sidekiq not running. Sidekiq heartbeat test failed, restarting

Wondering if I can resolve this issue by either flushing Redis or updating it; it hasn’t really been touched in the last 8+ months. I have not personally dealt with Redis before, but our Tests-Pass Discourse instance was setup using local_discourse rather than the Discourse Docker that is now recommended on this forum. Issues I’m encountering are related to email not being sent to users. Any help and suggestions appreciated!

Sidekiq is not running. Many tasks, like sending emails, are executed asynchronously by sidekiq. Please ensure at least one sidekiq process is running.

Sidekiq heartbeat test failed, restarting

config/unicorn.conf.rb:147:in `check_sidekiq_heartbeat
config/unicorn.conf.rb:164:in `master_sleep'
unicorn-5.5.4/lib/unicorn/http_server.rb:296:in `join'
unicorn-5.5.4/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'

If it makes sense to migrate, I’m open to suggestions.

I suggest taking a backup of your current install and creating a brand new one following the Discourse official Standard Installation

4 Likes

That’s been the recommendation for at least the last 4 years.

I’d second Rafael’s suggestion.

1 Like

Would it help for me to use launcher to rebuilt or restart the redis container?

Usage: launcher COMMAND CONFIG [–skip-prereqs] [–docker-args STRING]
Commands:
start: Start/initialize a container
stop: Stop a running container
restart: Restart a container
destroy: Stop and remove a container
enter: Open a shell to run commands inside the container
logs: View the Docker logs for a container
bootstrap: Bootstrap a container for the config based on a template
run: Run the given command with the config in the context of the last bootstrapped image
rebuild: Rebuild a container (destroy old, bootstrap, start new)
cleanup: Remove all containers that have stopped for > 24 hours
start-cmd: Generate docker command used to start container

Options:
–skip-prereqs Don’t check launcher prerequisites
–docker-args Extra arguments to pass when running docker
–skip-mac-address Don’t assign a mac address
–run-image Override the image used for running the container

Did you install following the Discourse official Standard Installation guide or some other way? If you have a standard installation, then yes, doing a ./launcher rebuild app will likely solve whatever your problem is.

Not yet, but we’re setting a time to migrate. Our Discourse has been running for almost three years and I just don’t want to add any unnecessary downtime. :slight_smile: Our instance is entirely maintained by volunteers so I’ve been teaching myself how to maintain it in the last month. If a quick redis fix will help (that container has been running for a year untouched) I’d like to apply it.

If you didn’t do a standard install to begin with, it’s anyone’s guess about what might fix your current problem.

I’d set up a new VM and do a test install there to make sure it works. When you do the transfer, you can put the old forum in read-only mode, take a backup, restore it to the new VM, and change the DNS for virtually no down time. (There will actually be a bit as you’ll need to rebuild after you change the DNS to get a let’s encrypt certificate installed)

1 Like

Thank you, the Let’s Encrypt should actually be quite easy since it has been setup to be a simple nginx reverse proxy edit. Good to hear, and expect, there will be some downtime. Just need to schedule this on an off hour.

I’m about to migrate and restore the backup into official images. Is there anything I should know regarding clearing out backed emails, etc. that would spam the users once sidekiq, etc. is running properly. Thanks!

The sidekiq job queue won’t move over, so a lot of emails will be dropped, while digests will resume sending normally.

3 Likes

So, attempting to follow the documentation for installing official Discourse docker images and running into trouble. We concluded we are missing the redis container, mail-receiver container and data container we’d had previously in Docker.

Seems our previous installation includes:
app.yml
data.yml
mail-receiver.yml
redis.yml

This multi-container setup differs from basic installation instructions. I do have a backup of our old /var/discourse for reference.

FAILED                                                                                                                                                        │················································································································
--------------------                                                                                                                                          │················································································································
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 645 exit 1>                 │················································································································
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'                                                                                            │················································································································
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}                                   │················································································································
bbf0e57ac69f1febe2a5f149aa7e6e12541c3c23aaf199188fdf19d507254b58                                                                                              │················································································································
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.                                                   │················································································································
./discourse-doctor may help diagnose the problem.

Seems we missed the step ./launcher bootstrap data
./launcher start redis

[Mon 11 May 2020 12:53:20 AM UTC] Run reload cmd: sv reload nginx                                                                                             │················································································································
fail: nginx: runsv not running                                                                                                                                │················································································································
[Mon 11 May 2020 12:53:20 AM UTC] Reload error for :                                                                                                          │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Domains not changed.                                                                                                        │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Skip, Next renewal time is: Thu 09 Jul 2020 11:33:04 PM UTC                                                                 │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Add '--force' to force to renew.                                                                                            │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Installing key to:/shared/ssl/discuss.noisebridge.info_ecc.key                                                              │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Installing full chain to:/shared/ssl/discuss.noisebridge.info_ecc.cer                                                       │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Run reload cmd: sv reload nginx                                                                                             │················································································································
fail: nginx: runsv not running                                                                                                                                │················································································································
[Mon 11 May 2020 12:53:21 AM UTC] Reload error for :                                                                                                          │················································································································
run-parts: executing /etc/runit/1.d/remove-old-socket                                                                                                         │················································································································
Started runsvdir, PID is 626                                                                                                                                  │················································································································
ok: run: redis: (pid 636) 0s                                                                                                                                  │················································································································
chgrp: invalid group: ‘syslog’                                                                                                                                │················································································································
ok: run: postgres: (pid 639) 0s                                                                                                                               │················································································································
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.                                                                               │················································································································
rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]                                                                  │················································································································
supervisor pid: 640 unicorn pid: 667
1 Like

Okay, we’ve successfully restored our previous Discourse instance! Now seeing:

All outgoing email has been globally disabled by an administrator. No email notifications of any kind will be sent.

That’s correct, the importers set that so you don’t blast 50,000 emails after a migration. Turn email on carefully in your site settings.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.