Is there an API scope which covers operations relating to groups?

Are you looking for something like “manage groups” (optional group id)?

This scope would allow you to add and remove users from a group