Make groups invisible to the public

Is there a way to either make all groups hidden to everyone, or disable the hyperlink for groups a person is a member of on their own profile page?

We are using some groups that automate via SSO to identify the types of users and potentially gate certain areas or our community. We don’t mind if users can see that they are part of a group, but we don’t want them to be able to see this info for anyone else. If I make the groups HIDDEN and turn off the directory then they only show up on my own profile if I’m a member of a group. But if I click on that group name I then see the whole list.