Granular API key restriction missing a lot of endpoints

If you’re trying to limit who can add to the group can’t you just make a user that has access only to that group? Say configure the group so that group owners can manage members? You don’t even need api key restrictions as you have user restrictions, right?