I can reproduce this on my local test site. I am not certain if this is the expected behaviour or not, but I can see how it could cause problems. I think that removing a category from a group’s muted categories list should either automatically unmute the category for the group’s users, or give the admin an option to unmute the category for those users.