Did you check that your host isn’t blocking outbound connections to mail.domain.com port 465?
If your mailer expects forums@domain.com as your notification email, did you change that setting in Discourse?
Oh, but wait. You have Discourse configured to use port 587, but the screenshot that you provide shows port 465, so you might want to tell Discourse to use that port.
Did you look at Troubleshooting email on a new Discourse install?