Do you have this set on the other groups:
It’s under ‘Effects’ on Manage/Membership for each of the groups.
If it’s set on that group and no others then it shouldn’t override.
The primary group will determine the default flair and title for the user.