You can do this, just make your groups, then when making or editing a category remove “everyone” and add the group(s) you want to have access instead under Security.
I do not believe they can be given to a specific person, however it could be given to a group with one person in it. (Under Settings > Moderation)