I think the only way to do this would be to create a group that includes everyone who is allowed to be in the category.
It would be nice to have a negative category permission available on category security settings, e.g. “Cannot create/reply/see”. But that does not exist now.