With the setting of the group visibility you decide who is able to see the group on the groups overview. They cannot read messages send to a group they aren’t a member of.
Whether a category is visible to the user, depends on the caregory security settings. If the user is not member of a group with “see” permission they will not see the category or any topic in it.
I would do that with Category Previews