I think that this is now working just fine.
I did notice something that isn’t great for my use-case, but is probably intended:
When I make the Tag Group visible to only a specific group, the tag does not appear in the watched list of those who can’t see it (nor does it apply). I used to use the tags to ‘bump’ the topics up to a higher watching state without regular users being able to see it. I can’t do that any more but that is fine.