Allow Category Moderator to add or remove Users from that Category

How are you handling this today?

Are you already making those individuals owners of the groups that have access to those categories?