Hi Ruth, you can not do what you asked for, but there is a possible workaround.
Create a parent category “Foo”, edit the permissions so everyone can see, create a subcategory of Foo, let’s say “Bar” and set the permission to your-custom-group can see-reply-create (or see-reply, it depends on your needs).
Everyone can see the Foo category, but only the custom group can open topic and see posts in the Bar subcategory