In the Discourse instance, a change to the default_watch_categories setting in version 3.1.0.beta5 unintentionally triggered a massive volume of emails—nearly 400K retries over three days, with over 9% failing due to SMTP provider rate limiting. Although the setting was reset, Sidekiq continued retrying failed emails due to the ongoing rate limits, causing persistent retries. The issue was resolved only after the problematic post was moved to a new topic, which cleared the erroneous notifications and stopped the retries. agemo highlights that the problem went unnoticed until SMTP limits caused failures, suggesting a need for better email activity monitoring and alerts. kris.kotlarek explains that the default email settings (e.g., only when away) may have contributed, and that Sidekiq jobs persist even after the triggering condition is removed. agemo confirms the default email level was set to only when away and never for message level, indicating a misconfiguration in notification logic.
default_watch_categories was set from default (which is nothing iirc) to a single category, probably for the first time ever.
This triggered a massive volume of emails, over a period of 3 days almost 400K email reties may have been processed, over 09% failing. Very early on this triggered rate limiting on the smtp providers side (which saved a massive unintended email bill!).
The default_watch_categories setting was reset as it was set inadvertently.
However sidekiq was still showing a huge volume of email retires, because the smtp provider rate limits were still in effect, it kept trying, and failing.
On closer inspection, it looked as though it was now just one post continually triggering the same email over and over, in the hundreds and then thousands of reties, even though the default_watch_categories had been reset to default. What was going on?
Once that post and others below it were split into their own “new” topic for normal moderation reasons the email trigger and retries stopped dead as a bonus. No more retries.
As an extra insight - this only got detected when it failed, due to the smtp providers limiters, maybe some clearer readout in the dashboard of the days email send activity for last 7 days, 24 hours, and hour, with alerts for any spikes, may be welcome.
This kind of thing could easily get you into a lot of trouble financially.
This could have chewed up our years hosting costs in a couple of days had the provider not had rate limits kick in!
