A lot of the group features are tied to the group being visible to the public.
I have the opposite problem from @jomaxro in that I have groups available to the public (e.g., for directories of users) but I don’t want the groups to be visible directly on the /groups page.
One solution for both issues would be to have a setting on each individual group for “show on Groups page”.
This feature could still respect the permissions for accessing non-public groups. Users could see that were was a group on the /group page, but if they select the group, it could show the permission/error screen unless they are a member of the group.