Sidekiq jobs not being processed, queue building up

I have Sidekiq jobs building up in the queues without being processed. I got aware of this when Discourse notified me about >100k jobs in the admin dashboard.

In Sidekiq’s web interface I see this:

Job	                Arguments	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::RunHeartbeat	[{"current_site_id"=>"default"}]	
Jobs::DownloadBackupEmail	[{"to_address"=>"daniel@xltoolbox.net", "backup_file_path"=>"https://forum.xltoolbox.net/admin/backup...

The email entry was created when I attempted to download a backup; the link to the backup file was never mailed to me.

Discourse is working fine otherwise. The e-mail test succeeded. Discourse’s log shows nothing. I’ve read "Ensure sidekiq is running." when it is definitely running which prompted me to try to download a backup, but did not help to solve my problem.

This is with a non-Docker install on Ubuntu 14.04. I know this setup not supported, but maybe someone can point me in the right direction. Thanks.

While you probably won’t like this answer, the best way is to export a backup, reinstall Discourse using these instructions, and restore your backup.

3 Likes

I think I finally found the cause. Sidekiq is started as a custom upstart job, and I had failed to tell Sidekiq what queues to process (with -p critical -q default -q low – although I don’t even know if the default queue is really needed). Thanks to this StackOverflow answer and the Sidekiq reference.

Maybe this helps others with a similar problem. But I guess if you don’t use the Docker image, you’re bound to run into problems like this one.

5 Likes

I opened this PR for those wanting to reference sidekiq.yml rather than pass explicit command line arguments.

2 Likes