Sidekiq is unexpectedly paused

**MHR** · January 16, 2019, 12:07pm

We have a setup for both Topic Event / Post Event webhooks, which was working fine.
But suddenly it stopped working and on any Topic Event or Post Event no webhook was sent.

We have checked for Discourse log also but there is no error.

So is there any way we can troubleshoot this issue?

**pfaffman** · January 16, 2019, 7:57am

Check the sidekiq queue. Is it filed up with posts being rebaked due to the recent image changes?

**MHR** · January 16, 2019, 9:23am

Thanks for your reply,
I found all my web hook events in scheduled list in sidekiq.

Can you please help me with below things?

How to fix it?
what does schedule list means? and How to execute those web hook now?

**MHR** · January 16, 2019, 9:37am

Finally found the issue.
Sidekiq was paused.

**codinghorror** · January 16, 2019, 12:07pm

Why was it paused? Did you pause it?

**MHR** · January 16, 2019, 1:03pm

Not sure but may be it’s because of failure of backup.

**pfaffman** · January 16, 2019, 2:21pm

Make sure you have disk space and reboot.

**codinghorror** · January 16, 2019, 10:27pm

What happens is sidekiq is paused during backup, but if backup fails, sidekiq is never unpaused. Didn’t you make a bunch of improvements here recently @tgxworld?

Are you on the latest version of Discourse @mhr?

**tgxworld** · January 21, 2019, 3:46am

We’ll always unpause Sidekiq even if the backup/restore fails.

**MHR** · January 21, 2019, 1:09pm

we are currently using v2.1.0.beta6 +89

**codinghorror** · January 21, 2019, 1:12pm

You need to update to latest.

**codinghorror** · February 10, 2019, 4:47am

This “pause sidekiq during backups” decision is the source of considerable trauma @tgxworld. Especially for giant backups going to slow offsite network dumps, or if something goes wrong with the backup and it never “finishes”, etc… users don’t get notifications and so on because Sidekiq is paused, and the longer this goes on… the worse the effects are on users.

Why did we make this decision to pause Sidekiq during backups? do you remember @sam?

**pfaffman** · February 10, 2019, 5:20am

One reason is that if sidekiq is running, the backup can fail because an upload gets deleted when tar is running. But maybe having a backup fail isn’t that bad.

**sam** · February 10, 2019, 6:40am

Previous to this fix we would have backups often fail due to transaction deadlocks with sidekiq jobs.

The intention here was always to simply pause sidekiq, run pgdump, unpause sidekiq, create tar, compress tar, upload backup

It is very possible there are some flow bugs we need to iron out, seeing issues in our hosting as well, will review this next week

**sam** · February 18, 2019, 4:40am

I just committed an awesome fix by @tgxworld for multisite that makes this more reliable. @gerhard also reduced the sidekiq pause window!

We have some additional work left here (never allow you to pause for longer than N hours) so assigning this to @tgxworld till it is implemented.

**tgxworld** · February 21, 2019, 5:55am

A change was made to the backup process which makes it extra sure that the sidekiq process will be unpaused.

github.com

discourse/discourse/blob/46e62c0d22099bbdb86d98c9e78fa5687e7777f5/lib/backup_restore/backuper.rb#L35-L41


begin
  pause_sidekiq
  wait_for_sidekiq
  dump_public_schema
ensure
  unpause_sidekiq
end

Closing this for now but feel free to flag this post if it happens again.

**tgxworld** · February 21, 2019, 5:55am