I figured out what appears to be the solution: Delete all of the access groups and get,
Then add the user groups desired, no one else will be able to see the category,
At least, that is how it appears to work. I haven’t tested. This is the feature I sought here. Thanks all.