This is an issue with the mail server certificate. You can probably get one with Let’s Encrypt, but it’s not a problem with Discourse, so ask for help from the docker-mailserver people. If you want people to receive your mail and not to spend a long long time getting your mail server trusted, you’ll be better off using a mail service like Mailgun.