Well, it would be nightmare for admins quite fast. Outsourcing administration to users is always bad idea.
Meaning an admin can create groups and let an user lead it. And create a category that is limited by that group and give rights of category moderation to that user.