Yes, I think that using the current setup, the easiest approach would be to not enable Single Sign On (SSO) between your WordPress site and the forum. This means that the forum and the website would manage users separately. When someone signs up for a course you would send them an invitation from the forum to join the associated discussion group. This groups content on the forum could be protected using the group/category setup. The links to that groups discussions on the WordPress site would also be protected by your membership plugin.
I think there is a small change we could make to the wp-discourse plugin that would make it possible to do something similar to this while using Single Sign On. The change would allow a WordPress user to be added to a Discourse group, by an admin, from the WordPress site. Something like this should work quite well with any membership plugin.
Complete integration between Discourse and a membership plugin is possible. This would automatically create a Discourse user and add them to a group when they sign up for a membership. The problem is that every membership plugin is different, so code would have to be written specifically for that plugin. I have started looking at doing this with the MemberPress plugin (because the people who make it gave me a copy of it to work on.)