Topic permissions are done at the category level. The easy solution is to do things that way.
Maybe check out Private Topics Plugin and Topic replies invisible until topic owner decides to reveal them? - #10 by riking. They aren’t what you need, but they are examples of plugins that violate the “permissions are done at the category level” rule.