Sidekiq is unexpectedly paused

(Mihir) #1

We have a setup for both Topic Event / Post Event webhooks, which was working fine.
But suddenly it stopped working and on any Topic Event or Post Event no webhook was sent.

We have checked for Discourse log also but there is no error.

So is there any way we can troubleshoot this issue?

(Jay Pfaffman) #2

Check the sidekiq queue. Is it filed up with posts being rebaked due to the recent image changes?

(Mihir) #3

Thanks for your reply,
I found all my web hook events in scheduled list in sidekiq.

Can you please help me with below things?

  1. How to fix it?
  2. what does schedule list means? and How to execute those web hook now?
(Mihir) #4

Finally found the issue.
Sidekiq was paused.

(Jeff Atwood) #5

Why was it paused? Did you pause it?

(Mihir) #6

Not sure but may be it’s because of failure of backup.

(Jay Pfaffman) #7

Make sure you have disk space and reboot.

(Jeff Atwood) #8

What happens is sidekiq is paused during backup, but if backup fails, sidekiq is never unpaused. Didn’t you make a bunch of improvements here recently @tgxworld?

Are you on the latest version of Discourse @mhr?

(Alan Tan) #9

We’ll always unpause Sidekiq even if the backup/restore fails.

(Mihir) #10

we are currently using v2.1.0.beta6 +89

(Jeff Atwood) #11

You need to update to latest.

(Jeff Atwood) #12

This “pause sidekiq during backups” decision is the source of considerable trauma @tgxworld. Especially for giant backups going to slow offsite network dumps, or if something goes wrong with the backup and it never “finishes”, etc… users don’t get notifications and so on because Sidekiq is paused, and the longer this goes on… the worse the effects are on users.

Why did we make this decision to pause Sidekiq during backups? :thinking: do you remember @sam?

(Jay Pfaffman) #13

One reason is that if sidekiq is running, the backup can fail because an upload gets deleted when tar is running. But maybe having a backup fail isn’t that bad.

(Sam Saffron) #14

Previous to this fix we would have backups often fail due to transaction deadlocks with sidekiq jobs.

The intention here was always to simply pause sidekiq, run pgdump, unpause sidekiq, create tar, compress tar, upload backup

It is very possible there are some flow bugs we need to iron out, seeing issues in our hosting as well, will review this next week