Instead of sending an email to the contact email address, we changed it so it sends a private message to all admin and staff.
The setting’s description is out of sync with this new behaviour.
If you want to get emailed about every staff message like that, you should change your notification level from “Tracking” to “Watching”.
But a problem with this whole thing is that the check is only happening once a day, even if you set the setting to 1. It should be improved to check hourly and only create a new notice if there are new flags.