This is the magic. Adding to a group that lacks the flag doesn’t assert removal of the flag - that would be a nightmare to maintain…but retroactively applying the flag and then removing the flag to the group works as a decent workaround if I only have to do it annually-ish.
#feature-request - ability to assert a ‘nullify’ category or tag on group add would be sweet.
