SSO groups without completely overriding

What do you mean by “this”, groups or the individual add_groups and remove_groups ?

What are the exact params and endpoint you are calling?