Hi Marc
So everyone can see the category? Just not create/reply when not in the group? If this is the case, I think a workable solution would be this:
In the category settings, scroll down and find this
Then add something like this
You must be part of the <a href="https://your-site-name/g/group-name-here">Group-name-here</a> group to create topics in this category
This will display a small banner to users who are not in the group. Once they join it disappears.