Disable E-Mails set to non-staff but not sending out test email or administrator verification

We have Disable E-Mail set to non-staff in the settings but what happens seems to be no mails get send out at all. The test feature in the setting does send no mail and so does the system when I try to make a user an admin. The verification email doesn’t seem to get send.

I’m very confident about my email settings as I used it when installing discourse. I send some mails successfully (it’s Amazon SES so I assume it should still work).

At least the administrator verification email should be sent I guess as it’s a staff email, isn’t it?

EDIT: E-Mails are sent when I try to change the email address on my staff account from my profile so I assume those admin verification emails are just not included in “non-staff” which I think they should.