Sorry. That’s a strange request.
Yes, you can give access to a subcategory to a group. No, you can’t give a group rights to make a topic a wiki. You could configure the category so that topics are wikis and make the permission so that only that group can create topics.