That’s interesting. We also have that setting checked on our site but it seems that moderators do not always get the alert in the Review Queue.
I thought it was perhaps a quirk related to our particular Discourse instance but perhaps not.
The problem is that it is completely random; sometimes we get alerts, other times not.
Any ideas how to troubleshoot this?