Regression causing erroneous, incorrect email notifications sent to all users in the approval queue for over 60 days

Bug
1

In Discourse, you can specify a mode where user accounts are created on signup, but must be approved by a staff member before the account is activated. This is common on subscription based sites, for example.

There were two important aspects of how this worked:

  1. No negative notifications were ever sent when staff rejected accounts, only a single positive notification was sent when staff approved accounts.

  2. Any item in the review queue, whether it is a post or an account, will be default-denied after 60 days if a moderator does not act on the review queue item in that 60 day period. This has existed in many previous versions of Discourse, as this path was chosen to be “default safe”.

Unfortunately, when we added a new, requested feature to let users know why they were rejected, what we did not realize is the combination of #1 and #2, above, would cause Discourse to send individual negative notifications to all users who existed in the approval queue for more than 60 days!

Those email notifications looked like this:

image

and said, under the customer’s brand,

A staff member rejected your account on {brand}.

This message being sent to so many “rejected” accounts has caused problems for our hosted customers. We’re deeply sorry about this mistake – it is completely our fault at Discourse, and all the blame should lay with us.

We’re currently working with our customers to do everything we can to make it right.

We’d like to be crystal clear that this is our bug and we sent a very confusing, badly worded email to a lot of their customers, resulting in considerable disruption, and possibly even loss of revenue as customers cancel their subscriptions. We’re so sorry about this, and the very least we can do is to let everyone know that it’s a problem with Discourse, we at Discourse incorrectly sent that negative email. We’re fully responsible for that bug and its consequences. If you received that email, it was through no fault of anyone but us – at Discourse.

We understand how serious this is. We continue to work closely with the membership provider, and our affected customers, to take responsibility for our actions and improve things so this can’t happen again.

We’re also offering the next month of this customer’s hosting free, as compensation for the problems we’ve caused. We sincerely apologize for the error and the way it affected our customers. We will absolutely be revisiting our processes to make sure this type of problem is less likely to occur in the future.

Jeff Atwood
CEO, Discourse

28 Likes
4

Here’s what we did to fix the issue and prevent it from happening again:

https://github.com/discourse/discourse/commit/f75e1867ff44a8c955994f8f4bc94af681086ebd

This commit is a 3-in-1 as it fixed 3 issues that led to this bug and prevented us from quickly finding the root cause.

  1. We made sure that the “email logs” are not deleted when a user account is deleted. If we had email logs of deleted users, we could have quickly seen all those rejection emails that were sent.

  2. We ensured that everywhere we delete a user in the code, we properly set the “context” so we can clearly pinpoint where and why a user has been deleted. There were several places in the code where the context wasn’t set so we did not know why those users were deleted and had to dig in the code to figure it out.

  3. We fixed the actual bug and won’t send rejection emails to auto-deleted reviewable users anymore.

All of these changes were accompanied with tests so we can be sure we won’t regress.

18 Likes