These are the out of the box group permission sets that can be set for each Category by group:
You will note there is no way to prevent someone replying to a Topic they didn’t create unless they have only See capability.