I disagree. things like the weird behaviour of the back-button and no undo make this widget very error-prone in our daily work (aka accidentally deleting people without even noticing). I’d separate the concern of group settings from the member management.
I see the point of that being hidden and changing the user view making it complex though. So I amended the Spec to include a simplified user-list on the bottom of the group instead of the current widget and make the top part only about the settings. This version would not amend the User-Listing-View (because of previously mentioned complexity problems) and of course you can’t remove or add on automatic lists. But I think it is much cleaner and easier to follow – and through endless-scrolling shouldn’t be limited to anything ever.