Indeed; the minimum capability a group can have is “See”. If you don’t want a group to be able to see the category, then you need to remove the group completely from the permissions.
There is no way to add a group with no “See” rights, it would be pointless.