You can’t have topic-specific permissions in a regular category, no.
The best way to do this is using a ‘group inbox’. There are a few descriptions of how this works - here’s one comparison of categories and group inboxes:
We describe how ‘team Discourse’ use group inboxes in this blog post: