Sorry. I’ve done my best here and have no more ideas.
If you want me to solve your problem, then you can see my installation packages: https://www.literatecomputing.com/discourse-installation-packages/.
I can configure Mailgun for you and it will work. There are a hundred different things that might be causing your custom mail server not to work.