Hosting my own mailserver for Discourse?

This site has a good tutorial on setting up your own ISP-level mail server:

https://workaround.org/ispmail

All I’d say is that it’s not trivial, but the tutorial and the introduction should give you an idea on what’s necessary.