Sorry for the late response here but I have been exploring nextcloud and discourse integration options recently. You’ve probably figured all this out by now but I’d be interested to compare notes. Are you still using discourse and nextcloud together and if so can you share some more info on what you’re trying to achieve and how it’s working out for you? I’d appreciate it!
Yes, as you have learned, Discourse does not like to be loaded in an iframe. It has always been this way. The workaround is to use Discourse’s own very flexible theming system to add a header that matches your main site. You can also then use nextcloud’s theming system to update it to match, as you see fit. In the nextcloud external sites admin setting, you’ll then select the redirect option meaning the site does not allow embedding. The link to discourse will then open in the same browser tab, without the iframe.
The Nextcloud discoursesso app you refer to is just about allowing people to authenticate to your discourse site via Nextcloud. Looks like you are setting it up correctly. You just need to go to the URL of your discourse and try to log in, and you will see you are logged in using your nextcloud credentials.
Note that in Discourse, Discourse SSO has been renamed to Discourse Connect. Enabling it will totally replace any other login options in your discourse so you will want to use it carefully.