I must admit that yes, there are a few slight confusions.
I would say it’s like in “real life”: Of course, members of a group can talk to each other. But can we make it so they can ONLY talk to each other? (i.e., prevent them from talking to non-members of the group)? I don’t think so.
I believe we need to see things exactly as described above: A group = people who are members or not, without this having any real consequence IN ITSELF. For example, the group can be mentioned using “@group” (“at sign group”).
Then, there is private and public. Everything that happens privately is not visible to others. Note that a private message can include multiple people.
What you clearly want is this:
Create categories reserved for members of a group. You act at the level of categories that group discussion threads, not at the level of the group members themselves. However, membership in the group is what then grants access to these reserved categories.
Hope this is fairly clear.
(EDIT: Unless this is what you’ve already done, and things still appear in “activity” when they shouldn’t? Does a non-member of the group have access to the messages/discussion threads you wish to keep reserved for the group? If yes, the issue isn’t the display of “activity,” but your settings.)