Permissions in certain category

You can check out Discourse Private Replies which sort-of (but not quite) does what you want. You might be able to modify or pay someone to modify that plugin to do what you want.

I think that using a theme component to hide the reply buttons to all but staff and author might be an easier fix, but don’t know for sure.