Avatar flair only appears if the group is the primary group of the user. Each user account can only have one primary group at a time, whichever was set as primary last wins.
Conversely if you want it to appear on the group but not on members make sure no users have it set as their primary.