Yeah, this is basically the reason why it works like this.
If you want to gate the entire feature to members of those 17 groups thatโs doable via the site setting, but the one time invite is per user (too easy to become spam amplification) and the room membership itself is per user (keeping it sync with groups would be a future improvement).