I think this is for performance reasons:
Could you create a specific group for your category moderators?