Sorry in my haste I didn’t notice one restriction you included:
In a Category there’s no way to prevent someone from replying to a Topic they did not create.
‘Reply’ permission covers an entire Category and all its Topics
Out of the box there is no way to do what you want to achieve.
A plugin could achieve that. Marketplace is your friend 