I just tried testing this with Discourse Version - 2.9.0.beta3, however, I was unable to reproduce the issue.
Specially, I setup a tag group with settings similar that what you described, ie - “Tags are visible for everyone, but only the following groups can use them” and then selected two groups and clicked save.
After reloading, the two groups were both visible, and upon clicking the “Tags are visible to everyone…” menu, both groups were able to be deselected.
I used Chrome Version 99.0.4844.84 to access Discourse, and also tested with another admin on the site using Firefox 91.7.1esr and was still unable to reproduce the issue.
I hope this helps provide some insight into this bug.