Having looked into this more: I think this only applies to people signing up after receiving an invitation email, so that might be where the confusion lies. As in, I think the setting was intended only for public registrations, but people like me assume it applies to all registrations.
It would be great if we could stop this happening with people who get invited via email too, but if not, I guess a workaround is just to remove the ability for anyone to be invited via email.