I think what you’re describing is normal behavior for the site.
Instead of the 404 page showing on categories that a user does not have access to, you want a page that shows them what the category is and to be able to join the group to access it?
There’s no way to configure this in Discourse, but you could get someone to write a plugin for this in Marketplace. This isn’t a trivial effort, though, so you will need to have a reasonable budget for this.