Sounds like a networking issue. You’ll need to work it out with your firewall or hosting provider.
Then you’ll need to configure your mail server to allow Discourse to connect. It’s easiest (at least from the Discourse perspective) if you assign a password.
There are a zillion things that make running a mail server hard. FWIW, back when email was easy, I ported sendmail and UUCP to Linux and ran my own mail server for a decade. I have tried a couple times to set up a mail server and haven’t yet bothered to be successful.