Looks like the way that the public link used to work is broken. It s mostly never used. That should be deleted.
The smtp test does check if it’s able to connect, but doesn’t do dns checks.
That’s docker exec error suggests that something is wrong there.
I’ll see about adding a note about the sending address. That’s still a common problem. Discourse sends from the forum subdomain and often the mail server expects the apex domain…
For your email issue, have you checked Troubleshooting email on a new Discourse install?