You don’t necessarily need to remove the restrictions on the existing categories. If it is desirable to have a place for A-only discussions, you can still have that.
If a topic is created in one of those private categories and you later want to make it available to other departments, you can then simply move the topic to that miscellaneous category and tag as appropriate.