I agree with this approach - one that this less rigid than the traditional binary access / no access.
It is often useful to have things freely accessible to those that dig for it, but only highly visible / notifying to a select group of people. This is pretty easy to initiate in Discourse, but hard to retract - I believe that this is what you are looking to do.
Users can of course do this themselves, but are generally pretty useless on that front.
Unfortunately, the Custom Wizard Plugin 🧙 requires user interaction to initiate stuff like that. An automation (likely bespoke) would be needed to remove the watching status of those that are no longer in a group.
Or this needs to be a feature request to alter the core behaviour. There are a few feature requests along these lines already.