I’m afraid what you have there is probably as far as I can really assist without knowing more about your server and the configuration. (kernel, apache, mods, docker, etc.)
I’m already making assumptions about your server that may or may not be correct.
To be more clear, I’m not telling you how to get what you want to happen done. I’m simply pointing you in the right direction. (Helping you to know the right question to ask since “wordpress + discourse” is a rather ambiguous question.)
Yes, you can have both on the same server. No, you’re not gonna be able to use both webservers on port 80. What you really should be looking for, is a way to serve up one app on the subdomain, and serve up the other on the parent domain. Your best bet is to configure this directly with Nginx or Apache. Since your site is Wordpress first, you may want to use Apache to handle this.
In summary: Nginx and Apache are both trying to listen on port 80 and they can’t both do that. Have one listen on another port, then proxy that connection through the subdomain to the client. How you decide to handle that is up to you, but will depend a lot on your specific situation, current configuration, VPS instance and preferences.