Show categories to everyone except a specific group

You should say more about your specific situation.

Categories that are visible to everyone are visible to everyone. You want to violate a tautology.

It sounds like what you really want is a separate forum for this other group.