Configure direct-delivery incoming email for self-hosted sites

pfaffman · December 21, 2022, 7:42pm

You can’t use https be because you didn’t uncomment these:

Mathias Foster:

# uncomment these (and the volume below!) to support TLS
#  POSTCONF_smtpd_tls_key_file:  /letsencrypt/discourse.example.com/discourse.example.com.key
#  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/discourse.example.com/fullchain.cer
#  POSTCONF_smtpd_tls_security_level: may

Simon_Manning · December 21, 2022, 10:10pm

That only applies to TLS support on the mail server side, i.e. for other mail servers to be able to deliver emails to mail-receiver over TLS.

Worth doing since the Discourse container evidently does have a certificate but shouldn’t affect mail-receiver connecting to Discourse. Potentially the rebuild could though if it happens to correct something in the container’s networking.

MathiasFoster · December 21, 2022, 10:47pm

Thank you, I uncommented those lines, and the line for the volume.

My mail-receiver.yml file now looks like this:

root@forum:/var/discourse# cat containers/mail-receiver.yml
## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:release
update_pups: false

expose:
  - "25:25"   # SMTP

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8

  ## Where e-mail to your forum should be sent.  In general, it's perfectly fine
  ## to use the same domain as the forum itself here.
  MAIL_DOMAIN: forum.[domain].co.nz
# uncomment these (and the volume below!) to support TLS
  POSTCONF_smtpd_tls_key_file:  /letsencrypt/forum.[domain].co.nz/forum.[domain].co.nz.key
  POSTCONF_smtpd_tls_cert_file:  /letsencrypt/forum.[domain].co.nz/fullchain.cer
  POSTCONF_smtpd_tls_security_level: may


  ## The URL of the mail processing endpoint of your Discourse forum.
  ## This is simply your forum's base URL, with `/admin/email/handle_mail`
  ## appended.  Be careful if you're running a subfolder setup -- in that case,
  ## the URL needs to have the subfolder included!
  DISCOURSE_MAIL_ENDPOINT: 'https://forum.[domain].co.nz/admin/email/handle_mail'

  ## The master API key of your Discourse forum.  You can get this from
  ## the "API" tab of your admin panel.
  DISCOURSE_API_KEY: '074[rest of API key - yes I generated a new one limited to the system user]d98'

  ## The username to use for processing incoming e-mail.  Unless you have
  ## renamed the `system` user, you should leave this as-is.
  DISCOURSE_API_USERNAME: system

volumes:
  - volume:
      host: /var/discourse/shared/mail-receiver/postfix-spool
      guest: /var/spool/postfix
# uncomment to support TLS
  - volume:
      host: /var/discourse/shared/standalone/letsencrypt
      guest: /letsencrypt

When I send a new email and run ./launcher logs mail-receiver, here is what I see:

Dec 21 22:41:21 forum-mail-receiver postfix/smtpd[132]: connect from mail-pj1-f54.google.com[209.85.216.54]
Dec 21 22:41:23 forum-mail-receiver postfix/smtpd[132]: 16DAC379E42: client=mail-pj1-f54.google.com[209.85.216.54]
Dec 21 22:41:23 forum-mail-receiver postfix/cleanup[139]: 16DAC379E42: message-id=<94fc2bef18b410ae8b121c6af2da2df4@frontapp.com>
Dec 21 22:41:23 forum-mail-receiver postfix/qmgr[100]: 16DAC379E42: from=<[my email address]>, size=5585, nrcpt=1 (queue active)
<23>Dec 21 22:41:23 receive-mail[141]: Recipient: nobody@forum.[domain].co.nzDec 21 22:41:50 forum-mail-receiver postfix/smtpd[143]: connect from mail-oa1-f50.google.com[209.85.160.50]
Dec 21 22:41:52 forum-mail-receiver postfix/smtpd[143]: 2E445379E48: client=mail-oa1-f50.google.com[209.85.160.50]
Dec 21 22:41:52 forum-mail-receiver postfix/cleanup[139]: 2E445379E48: message-id=<6b2f9d646dc46f4fec4af006de01d3ae@frontapp.com>
Dec 21 22:41:52 forum-mail-receiver postfix/qmgr[100]: 2E445379E48: from=<[my email address]>, size=4100, nrcpt=1 (queue active)
<23>Dec 21 22:41:52 receive-mail[147]: Recipient: nobody@forum.[domain].co.nzDec 21 22:41:53 forum-mail-receiver postfix/smtpd[132]: disconnect from mail-pj1-f54.google.com[209.85.216.54] ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
Dec 21 22:41:58 forum-mail-receiver postfix/qmgr[100]: 1194937A670: from=<double-bounce@forum-mail-receiver.localdomain>, size=942, nrcpt=1 (queue active)
Dec 21 22:41:58 forum-mail-receiver postfix/smtp[149]: fatal: unknown service: smtp/tcp
Dec 21 22:41:59 forum-mail-receiver postfix/qmgr[100]: warning: private/smtp socket: malformed response
Dec 21 22:41:59 forum-mail-receiver postfix/qmgr[100]: warning: transport smtp failure -- see a previous warning/fatal/panic logfile record for the problem description
Dec 21 22:41:59 forum-mail-receiver postfix/master[1]: warning: process /usr/lib/postfix/sbin/smtp pid 149 exit status 1
Dec 21 22:41:59 forum-mail-receiver postfix/master[1]: warning: /usr/lib/postfix/sbin/smtp: bad command startup -- throttling
Dec 21 22:41:59 forum-mail-receiver postfix/error[150]: 1194937A670: to=<postmaster@forum-mail-receiver.localdomain>, orig_to=<postmaster>, relay=none, delay=1192, delays=1191/1/0/0.01, dsn=4.3.0, status=deferred (unknown mail transport error)
<19>Dec 21 22:42:23 receive-mail[141]: Failed to POST the e-mail to https://forum.sobercheck.co.nz/admin/email/handle_mail: execution expired (Net::OpenTimeout)<19>Dec 21 22:42:23 receive-mail[141]:   /usr/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `open'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
  /usr/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
  /usr/lib/ruby/2.7.0/net/http.rb:958:in `connect'
  /usr/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
  /usr/lib/ruby/2.7.0/net/http.rb:932:in `start'
  /usr/lib/ruby/2.7.0/net/http.rb:1483:in `request'
  /usr/local/lib/site_ruby/mail_receiver/discourse_mail_receiver.rb:43:in `process'
  /usr/local/bin/receive-mail:13:in `<main>'Dec 21 22:42:23 forum-mail-receiver postfix/pipe[140]: 16DAC379E42: to=<nobody@forum.[domain].co.nz>, relay=discourse, delay=60, delays=0.23/0.01/0/60, dsn=4.3.0, status=deferred (temporary failure)
Dec 21 22:42:25 forum-mail-receiver postfix/smtpd[143]: disconnect from mail-oa1-f50.google.com[209.85.160.50] ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
<19>Dec 21 22:42:52 receive-mail[147]: Failed to POST the e-mail to https://forum.[domain].co.nz/admin/email/handle_mail: execution expired (Net::OpenTimeout)<19>Dec 21 22:42:52 receive-mail[147]:   /usr/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `open'
  /usr/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
  /usr/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
  /usr/lib/ruby/2.7.0/net/http.rb:958:in `connect'
  /usr/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
  /usr/lib/ruby/2.7.0/net/http.rb:932:in `start'
  /usr/lib/ruby/2.7.0/net/http.rb:1483:in `request'
  /usr/local/lib/site_ruby/mail_receiver/discourse_mail_receiver.rb:43:in `process'
  /usr/local/bin/receive-mail:13:in `<main>'Dec 21 22:42:52 forum-mail-receiver postfix/pipe[146]: 2E445379E48: to=<nobody@forum.[domain].co.nz>, relay=discourse, delay=60, delays=0.15/0.01/0/60, dsn=4.3.0, status=deferred (temporary failure)
Dec 21 22:45:45 forum-mail-receiver postfix/anvil[135]: statistics: max connection rate 1/60s for (smtp:209.85.216.54) at Dec 21 22:41:21
Dec 21 22:45:45 forum-mail-receiver postfix/anvil[135]: statistics: max connection count 1 for (smtp:209.85.216.54) at Dec 21 22:41:21
Dec 21 22:45:45 forum-mail-receiver postfix/anvil[135]: statistics: max cache size 2 at Dec 21 22:41:50

I’m really stuck now, does anyone have any ideas what could be causing this?

pfaffman · December 22, 2022, 2:30pm

Doh! Yes. I conflated the TLS and the https.

MathiasFoster · January 8, 2023, 11:32pm

This still isn’t working at all, no emails are getting passed on from mail-receiver to Discourse.

Can I ‘undo’ mail-receiver back to the start (reset it entirely) and start again, in the hopes that it will work?
How can I do this?

pfaffman · January 9, 2023, 3:07am

You can just edit the file and rebuild the mail container.

jryans · January 26, 2023, 10:52am

Thanks for the tip about the firewall! I also encountered troubles similar to @MathiasFoster, with the mail-receiver container unable to reach the forum site in the app container. A bit puzzling at first, since containers are being allowed to listen to the outside world without issue.

I am also using Vultr as my VPS provider with their Ubuntu OS image. Some combination of the OS image defaults plus Docker does indeed seem to block communication across containers.

Anyway, in my case, it was enough to allow HTTPS on the host:

$ ufw allow https

After that, the mail-receiver was able to deliver mail as expected.

saulshanabrook · February 16, 2023, 9:22pm

Does anyone have reccomendations for online hosts which don’t block port 25?

I am hosting my discourse server locally on my home ISP, but they block it… Ideally, maybe there is some service out there I could point my domain at which would forward any SMTP traffic to my server on a different port? But I don’t think that exists…?

pfaffman · February 16, 2023, 10:23pm

At least most hosting services (like Digital Ocean) do not block incoming port 25.

Virtually all home ISPs block port 25. If you want incoming mail on you home-based internet, you’ll need to retrieve it via POP from some mailbox.

saulshanabrook · February 16, 2023, 11:04pm

I believe Digital Ocean does block port 25 now:

SMTP port 25 is blocked on all Droplets for new accounts to prevent spam and other abuses of our platform.

I actually realized that SendGrid support a free incoming SMTP relay, so I will use that and write a simple server to take the incoming messages over HTTP and then send them to the discourse mail receiver over SMTP.

pfaffman · February 16, 2023, 11:07pm

I believe that refers to outgoing port 25, not incoming port 25. I haven’t had any trouble installing incoming mail receivers. I do a bunch of installations for people with new accounts.

saulshanabrook · February 17, 2023, 12:21am

Ah that’s good to know!

For anyone else hitting this, I got this working with sengrid, using a small relay server:

dfriestedt · April 14, 2023, 6:43pm

After we enable TLS do we need to also expose port 465 too? Looks like 25 is only exposed by default.

sebix1 · April 27, 2023, 10:43am

Mails are usually only received via Port 25, if transport-encrypted, then with STARTTLS. SMTPS (465) is deprecated.

sjjh · May 21, 2023, 3:54pm

I’m trying to set up incoming email for a self-hosted discourse instance behind a reverse proxy (http(s), SMTP). Public domain: public.example.com, host behind reverse proxy: internal.example.com. I followed this manual but are stuck potentially due to a certificate error. I’m using self-signed certificates for the internal encryption between reverse proxy and discourse containers. The mail container seems to have an issue with the self signed cert presented by the discourse container although it is a chained cert. What did I do wrong or how to debug the issue further?
The (relevant) log output of the mail-in container (./launcher logs mail-receiver) is:

May 21 15:34:06 internal-mail-receiver postfix/qmgr[101]: BA3E16FDE7: from=<foo@example.com>, size=3836, nrcpt=1 (queue active)
<23>May 21 15:34:06 receive-mail[113]: Recipient: nobody@public.example.com<19>May 21 15:34:06 receive-mail[113]: Failed to POST the e-mail to https://internal.example.com/admin/email/handle_mail: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (OpenSSL::SSL::SSLError)<19>May 21 15:34:06 receive-mail[113]:   /usr/lib/ruby/2.7.0/net/protocol.rb:44:in `connect_nonblock'
  /usr/lib/ruby/2.7.0/net/protocol.rb:44:in `ssl_socket_connect'
  /usr/lib/ruby/2.7.0/net/http.rb:1009:in `connect'
  /usr/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
  /usr/lib/ruby/2.7.0/net/http.rb:932:in `start'
  /usr/lib/ruby/2.7.0/net/http.rb:1483:in `request'
  /usr/local/lib/site_ruby/mail_receiver/internal_mail_receiver.rb:43:in `process'
  /usr/local/bin/receive-mail:13:in `<main>'May 21 15:34:06 internal-mail-receiver postfix/pipe[112]: BA3E16FDE7: to=<nobody@public.example.com>, relay=discourse, delay=0.39, delays=0.19/0.01/0/0.2, dsn=4.3.0, status=deferred (temporary failure)

The (relevant part of the) config file mail-receiver.yml of the mail container is:

env:
  POSTCONF_smtpd_tls_key_file:  /ssl/ssl.key
  POSTCONF_smtpd_tls_cert_file:  /ssl/ssl.crt
  POSTCONF_smtpd_tls_security_level: may

  DISCOURSE_MAIL_ENDPOINT: 'https://internal.example.com/admin/email/handle_mail'

volumes:
  - volume:
      host: /var/discourse/shared/standalone/ssl
      guest: /ssl

The private key ssl.key contains the private key of the (internal) server. The chained cert ssl.crt contains: server-cert + ca-cert (as a new user I’m not allowed to upload a file, thus currently cannot provide the ssl.crt)

Simon_Manning · May 21, 2023, 6:28pm

The smtpd_tls environment variables relate to the smtpd server, i.e. the part other mail servers will interact with when delivering emails. When it is then trying to deliver the email to the handle_mail endpoint on internal.example.com, whatever Ruby is using doesn’t trust your certificate authority and therefore can’t trust your self-signed certificate.

For this to work, I think you have two options. The first is to modify mail-receiver.yml in order to include your root CA certificate in the container, such that Ruby will trust it. I’m not sure off the top of my head what that would look like, but it would basically be the same as getting Ruby to trust a new CA on any Linux system except via that container yml file.

The other option is just to change DISCOURSE_MAIL_ENDPOINT from using internal. to public., causing it to connect via your proxy which presumably has a certificate it will be able to trust by default.

sjjh · May 26, 2023, 2:29pm

Thanks for you help, worked that way!
(initially, my naive me assumed, that all communication between the containers would happen directly between the two containers anyway)

mpalmer · May 31, 2023, 1:22am

Communication does happen directly between the two containers, it’s just that you configured your Discourse container to use HTTPS with a self-signed certificate, which changes the required method of communication.