Yes, I have tried refreshing the page and No, it doesn’t make a difference.
The thing is that the setting used to work differently in past. When I used to enable the “Set Group as Primary”, it used to automatically set it as primary for all the existing and new users. Now that’s broken in a way where the Group title of the primary group is applied correctly to all the members but not the flair. This seems to be a recent change that broke the setting. I remember it working correctly as expected back in Jan/Feb this year when I set up the groups on my forums.