This I cannot reproduce. If I have a tag that has only been used on a PM or a topic in a restricted category, a user without access to those topics does not see the tag included in the tag list.
The only unfortunate behavior I’m seeing at the moment is what I mentioned surrounding deleted topics.