You create a custom group and add members to it. See How to create private categories using category security settings.
If you have some external database that has the members in it, you can use the API or a plugin to add people to the group accordingly.