It is not possible to change the member visibility of automatic groups via the groups Manage UI. More specifically: the UI allows selecting a new value, and it cheerfully reports “Saved!” when Save Changes is pressed, but the request is silently ignored by the backend.
This looks like it is a regression that was introduced into the group_params() method of GroupsController in FIX: Add `errors` field if group update confirmation by udan11 · Pull Request #16260 · discourse/discourse · GitHub