Recategorizing to Contribute > UX as this is not a Contribute > Bug. Badges and profiles aren’t the same thing, there are lots of other places that user accounts can be surfaced even if their profiles aren’t visible.
Badges are a gamification tool - part of that means letting users see which badges exist and who else might have them. If you aren’t interested in the gamification aspect have you considered just disabling badges completely?