We have a setup for both Topic Event / Post Event webhooks, which was working fine.
But suddenly it stopped working and on any Topic Event or Post Event no webhook was sent.
We have checked for Discourse log also but there is no error.
So is there any way we can troubleshoot this issue?
Check the sidekiq queue. Is it filed up with posts being rebaked due to the recent image changes?
Thanks for your reply,
I found all my web hook events in scheduled list in sidekiq.
Can you please help me with below things?
Finally found the issue.
Sidekiq was paused.
Why was it paused? Did you pause it?
Not sure but may be it’s because of failure of backup.
Make sure you have disk space and reboot.
What happens is sidekiq is paused during backup, but if backup fails, sidekiq is never unpaused. Didn’t you make a bunch of improvements here recently @tgxworld?
Are you on the latest version of Discourse @mhr?
We’ll always unpause Sidekiq even if the backup/restore fails.
You need to update to latest.
This “pause sidekiq during backups” decision is the source of considerable trauma @tgxworld. Especially for giant backups going to slow offsite network dumps, or if something goes wrong with the backup and it never “finishes”, etc… users don’t get notifications and so on because Sidekiq is paused, and the longer this goes on… the worse the effects are on users.
Why did we make this decision to pause Sidekiq during backups? 
do you remember @sam?
One reason is that if sidekiq is running, the backup can fail because an upload gets deleted when tar is running. But maybe having a backup fail isn’t that bad.
Previous to this fix we would have backups often fail due to transaction deadlocks with sidekiq jobs.
The intention here was always to simply pause sidekiq, run pgdump, unpause sidekiq, create tar, compress tar, upload backup
It is very possible there are some flow bugs we need to iron out, seeing issues in our hosting as well, will review this next week
I just committed an awesome fix by @tgxworld for multisite that makes this more reliable. @gerhard also reduced the sidekiq pause window!
We have some additional work left here (never allow you to pause for longer than N hours) so assigning this to @tgxworld till it is implemented.
A change was made to the backup process which makes it extra sure that the sidekiq process will be unpaused.
Closing this for now but feel free to flag this post if it happens again.