Do you want to actually mute it, or just set the notifications back to “normal” ?
If it’s the latter, I would say this is actually a Contribute > Bug (or better: the implementation does not fit this use case, while it feels logical that it would).
When I set group X to watch category Y, and become a member of that group, I’m indeed automatically watching category Y. Category Y is copied to my “watching” categories. There is no trace of how it got there, we don’t know if it was there because I added it before or if it was there because of my group membership.
When I leave group X, I keep watching category Y. The category is not removed from my “watching” categories, as one would expect.
Changing this implementation would be quite an effort, so I guess this would be something for a plugin right now. Just have something trigger on group removal
with the :user_removed_from_group event, get the categories, remove them from the user preferences.