I’m encountering this issue at the moment too. But I’m also aware that it will add yet another layer of complexity to group config.
Building on this, it might be much simpler to have all categories and tags that are being ‘watched’ appear automatically in the sidebar.
However, I could see this causing some loss of power as there will be categories that folk might like to watch, but never visit unless they get a notification. They would be wasted space on the sidebar. But they could hide these if they are annoying.