Category permissions are additive. i.e.
See
See and Reply
See and Reply and Create
A lot can be done by setting permissions based on group membership. But it sounds like you are looking for a way to have the permissions be subtractive instead?
In other words instead of
Category - group members only, others restricted
you want
Category - others allowed, group members only restricted
If so, I know of no way to do this.