This can be configured with existing groups. After you change the setting of the Visibility dropdown, you need to click the Save Changes button. If the changes are not saving for you after clicking the Save Changes button please let us know.
The issue with the “Allow users to send membership requests” checkbox being greyed out is related to the visibility settings. That checkbox is only enabled if the group’s visibility is set to either “Everyone” or “Logged on users”. Possibly something could be added to the user interface here to explain to users why the checkbox is disabled.