There’s not a way to specifically set that threshold on the category, but you could use trust levels to do something similar.
So for example, you could change the site setting tl2 requires topic reply count to 10, and then in your category’s settings you could restrict topic creation to Trust Level 2 or higher.
More on trust levels here, if you’re unfamiliar: Understanding Discourse Trust Levels