Don't use sidekiq queue to process "problems have been found" email

Contribute Feature

AI-generated summary

A discussion is held on improving the resilience of alerting for “problems have been found” email notifications. LeoMcA shares an experience with random Sidekiq crashes, which were not immediately apparent due to the lack of proper monitoring. In one instance, a notification email got stuck in the Sidekiq queue, delaying its delivery.

To address this issue, LeoMcA proposes sending the email synchronously to the DISCOURSE_DEVELOPER email or contact email as soon as problems are detected. Additionally, a synchronously triggered webhook is suggested for sites with third-party monitoring setups, in case SMTP is not working.

In response, codinghorror questions whether Sidekiq automatically restarts after crashing, as no epidemic problems were seen in their hosting or self-installs. However, LeoMcA clarifies that this was not the case in their experience and argues that the proposed feature request stands alone, as it aims to minimize the number of potential failures between problem detection and alerting.

1

We’ve been seeing some random sidekiq crashes recently, hopefully fixed by FIX: update mini_racer corrects erratic segfaults · discourse/discourse@f662d11 · GitHub

One problem is there was nothing obvious that told us this had happened. We’re working on improving our monitoring setup so someone gets a phone call when this happens, but I expect most communities don’t/won’t have that sort of support.

After restarting sidekiq the most recent time, a “problems have been found” PM came through alerting me that sidekiq was down. The problem being, it wasn’t sent while sidekiq was down, because it got stuck in the queue!

As it happens, the other times sidekiq went down and I brought it back up there was no such PM. I’m not sure why that was. It’s possible this PM was alerting me instead to the fact an update check hadn’t been done because the queue was so saturated after bringing sidekiq back up after a weekend of it being down.

If an email was sent synchronously to the DISCOURSE_DEVELOPER email, or contact email, or whatever makes most sense as soon as problems are detected this would mean much more resilient alerting on sites that don’t have the time to build a fully-fledged monitoring setup.

…unless of course SMTP is also not working, so perhaps as well as this, there should be a synchronously triggered webhook for those instances that do have some sort of third-party monitoring set up.

3 Likes
2

I am not entirely following – doesn’t sidekiq automatically restart after crashing, regardless? We didn’t see any epidemic problems on our hosting, or in supporting self installs.

1 Like
3

It hasn’t been for us - which may point to it being another problem - we’ve been having to restart our containers to bring it back up. We’re still investigating.

I think this feature request stands alone, however. Taking as many things as possible that might go wrong out of the path from problem to alert seems like a good idea to me.