Setting up DISCOURSE_SMTP_ADDRESS: 'localhost'?


(Peter N Lewis) #1

I can’t get past the initial account creation as I can’t get discourse to send any emails.

I set up a new DO instance, and configured exim on it. I’ve tested it out and it can relay email correctly (transcript below).

It doesn’t need a password or authentication or TLS. But my configuration is not working:

  DISCOURSE_HOSTNAME: 'forum.keyboardmaestro.com'
  ##
  ## The mailserver this Discourse instance will use
  DISCOURSE_SMTP_ADDRESS: 'localhost'          # (mandatory)
  DISCOURSE_SMTP_PORT: 25                        # (optional)
  DISCOURSE_SMTP_ENABLE_START_TLS: false

  # DISCOURSE_SMTP_USER_NAME: user@example.com      # (optional)
  # DISCOURSE_SMTP_PASSWORD: p@ssword               # (optional)

The exim logs show now attempted connection at all, and I’ve tried resending the activation email multiple times with lots of guesses at configuration, and multiple “./launcher rebuild app” between each.

Any suggestions?

# telnet localhost smtp
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 forum.keyboardmaestro.com ESMTP Exim 4.82 Ubuntu Wed, 28 May 2014 04:30:20 -0400
helo whatever
250 forum.keyboardmaestro.com Hello localhost [127.0.0.1]
mail from: info@discourse.org
250 OK
rcpt to: peter@stairways.com.au
250 Accepted
data
354 Enter message, ending with "." on a line by itself
From: info@discourse.org
To: peter@stairways.com.au
Subject: I hatte smtp

test
.
250 OK id=1WpZFz-0002th-9D
quit
221 forum.keyboardmaestro.com closing connection

Email error. Unable to send out email
How to set no user/pass for smtp in discourse setup?
(Jeff Atwood) #2

I strongly, strongly suggest you use the recommended Mandrill setup.


(Jens Maier) #3

Discourse runs inside an LXC container. Its loopback network is separated from the host’s loopback, hence you can only connect to services running inside the container if you try to connect to 127.0.0.1 from within the container.

Depending on its configuration, Docker usually creates a docker0 bridge device with an IP address in the 172.17.* range (at least it does on my Gentoo box). You should be able to tell Exim to listen on that address and have Discourse connect there instead of localhost. Alternatively, let Exim listen on the host’s public interface and drop connections from the outside in the firewall.


(Peter N Lewis) #4

Mandrill really isn’t going to work for me. My intention is to replace my existing YahooGourps mailing list. It has 1500 members and 58 messages last week, which means roughly 350k potential messages/month, which makes for $67/month. That’s many times the cost of the DO server which could easily handle this load (on top of the web serving). Now many of those users might be on digest or whatever, but thus is balanced against wanting to expand the user base (the whole point of moving from a YahooGroup mailing list to Discourse), so it just doesn’t make sense to me.

If Discourse really will only work with Mandrill, fair enough, I’ll look elsewhere despite its promise, but for ten minutes setting up exim on a DO server as a send-only mailing server, I’d really just rather figure out what the required configuration is.


(Sam Saffron) #5

Definitely not the case. Discourse already works just fine with any SMTP gateway, you just need to point it at it.

Mandrill / sendgrid whatever are just the simplest way to set up a mail server. Setting up postfix or what have you so it works robustly can be fairly complex. Personally I would be happy to have this built into the image at least optionally. However even if it was there are a bunch of “surrounding” things you need to do to get mail working in a reliable way we can not control (DKIM, SPF and so on)


(Peter N Lewis) #6

@elberet ahh, thanks. I really don’t understand all the goings on with the whole Docker packaging, but I’m off to read up on it to see how its networking works. Failing that, you’re right, allowing the public interface and then having exim drop the connection if its wrong would work, though it just seems to be opening security holes unnecessarily for a local connection. But maybe its unavoidable.


(Sam Saffron) #7

You can also expose socket files via a shared folder. Barring that you need to use the private network.


(Jens Maier) #8

Here’s the five minute summary:

  1. Each container gets its own private lo device. No sharing is possible here.
  2. Each container also gets its own ethernet device. This is one part of a virtual ethernet device pair with the other part living in the host docker host.
  3. Docker creates a bridge device on the host and links all the virtual ethernet devices to it. Thus the host and all active containers can communicate over a private and completely virtual network; on my Gentoo box, Docker chose a 172.17.0.0/16 network address (iirc, can’t verify right now).
  4. The hosts private virtual IP address should be stable, containers’ addresses are most likely somewhat random.
  5. Containers get internet access via IP masquerading. Internet access to containers is handled via DNAT (configured with the expose rules in app.yml). Network access to the host from containers works as usual and goes through the normal INPUT chain.

(Peter N Lewis) #9

OK, so I thought perhaps what I could do was add a loopback interface on a private IP:

ifconfig lo:40 192.168.40.1 netmask 255.255.255.0 up

And then set DISCOURSE_SMTP_ADDRESS to 192.168.40.1, and set exim up to listen on port 25 of 192.168.40.1. Again exim works, I can telnet to the IP:25 and send an email, it relays it properly.

But still no dice. I don’t even get a hint that discourse has even tried to make a connection at all.

Is there any logging in discourse I can look at to see what it thinks is going on?

I presume this whole docker “boxing” thing buys something impressive, because it just seems to make this simple task miserable.


(Jens Maier) #10

The loopback device is special. Data sent through lo does not go through routing, so anything sent through lo will never ever leave the computer where it was generated. This also applies to Docker/LXC containers - these have their own loopback device, independent from the system that hosts them. The address you assign to the host’s and/or the container’s lo device has got nothing to do with this.

Have exim listen on port 25 on the docker0 bridge device and have Discourse deliver mails to that IP address. Don’t use localhost and don’t assign any non-local IP to lo. :slight_smile:


(Peter N Lewis) #11

I must be missing something. Even if I open the mail server up to the world, and then

vi containers/app.yml and change:

The mailserver this Discourse instance will use

#DISCOURSE_SMTP_ADDRESS: ‘192.168.40.1’ # (mandatory)
DISCOURSE_SMTP_ADDRESS: ‘forum.keyboardmaestro.com’ # (mandatory)
DISCOURSE_SMTP_PORT: 25 # (optional)
DISCOURSE_SMTP_ENABLE_START_TLS: false

and then:

./launcher rebuild app

(and even ./launcher stop app / ./launcher start app - is that necessary)

and then try again to log in, and click here to send me another activation email, I get nothing. No hint of an attempted login to the mail server.

I don’t get it. Either

  1. My changes to discourse settings are not working (am I configuring
    it incorrectly, or not restarting it properly?)
  2. Discourse cannot connect to the server its running on.
  3. Discourse is failing to even attempt to resend the activation email. *

So I tried changing the settings to send it to one of my other smtp servers. And still no evidence that it is making any attempt to send email, no indication the SMTP server is getting a connection at all.

So that eliminates 2. So either my changes to the settings are not changing the settings, or discourse is not resending the activation email like it says it is.

Or I am just completely lost.


(Peter N Lewis) #12

I’ll try that, but currently it appears that my changes to discourse aren’t being applied, so I’m very confused as to where I’ve gone wrong.


(Jens Maier) #13

Ah, that’s one thing I can’t help with. I don’t use the launcher script…


Emails with local SMTP
(Lee_Ars) #14

I run postfix on my physical server and simply have docker & discourse configured to send mail to the server’s proper hostname. This saves having to futz around with exactly what localhost means in which context.

I’ve got these lines in my app.yml container file:

DISCOURSE_SMTP_ADDRESS: mail.bigdinosaur.org
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: postasaurus@bigdinosaur.org
DISCOURSE_SMTP_PASSWORD: (password)

Then I’ve added the container’s bridged IP address to mynetworks in /etc/postfix/main.cf so that authentication attempts from that address to port 587 are allowed and the container is allowed to relay.


(Peter N Lewis) #15

Ahh, ok, so I now understand that ./launcher ssh app will get me into the virtual machine where I can poke around.

And then I found /var/www/disclosure/log

And it has this message in it:

Started POST "/users/peter@stairways.com.au/send_activation_email" for **** at 2014-05-28 14:03:00 +0000

ActionController::RoutingError (No route matches [POST] "/users/peter@stairways.com.au/send_activation_email"):
  config/initializers/quiet_logger.rb:10:in `call_with_quiet_assets'
  config/initializers/silence_logger.rb:25:in `call'
  lib/middleware/unicorn_oobgc.rb:95:in `process_client'

Processing by ExceptionsController#not_found as */*
  Rendered exceptions/not_found.html.erb within layouts/no_js (8.6ms)
  Rendered common/_special_font_face.html.erb (1.4ms)
  Rendered common/_discourse_stylesheet.html.erb (2.3ms)
  Rendered text template (0.0ms)
Completed 404 Not Found in 41ms (Views: 1.1ms | ActiveRecord: 2.1ms)

No real indication of what is going wrong though, but perhaps this means something to someone?


(Peter N Lewis) #16

OK, now this is an actual bug report. I managed to get Discourse email working enough to verify this.

If you create a new account, it sends you an activation email. If you then try to login again, it has a link you can click to send another activation email - clicking this link does not work and gives the error message described above. If you try to log in a third time, the link is not there, and although it says “We sent another activation email” it does not in fact do this (does not even appear to try again).

Given the fact that you cannot progress at all with your Discourse setup while email is not working, other than obviously fixing the above bug, it might be worth including a “send test email” button in with the message saying /var/www/discourse/script/test_email_settings.rb was useful in verifying that my configuration should be working, but it would be better if it could pick up the settings actually used inside Discourse.


(Peter N Lewis) #17

And to finish off this topic (hopefully), to set up disclosure to use exim on the same machine, you need to do the following:

In app…yml

DISCOURSE_SMTP_ADDRESS: ‘disclosure.example.com’ # (whatever your server name is)
DISCOURSE_SMTP_PORT: 25

Install exim as described at: How To Install the Send-Only Mail Server "Exim" on Ubuntu 12.04 | DigitalOcean

Adjust your exim settings /etc/exim4/update-exim4.conf.conf

dc_local_interfaces=''
dc_relay_nets='127.0.0.1:YOURDOIPADDRESS:172.16.0.0/12'

That last entry 172.16.0.0/12 allows sending from the private IPs on your server, one of which will be your discourse docker box.

Restart exim with:

sudo update-exim4.conf
sudo /etc/init.d/exim4 restart

Restart discourse if necessary with:

./launcher rebuild app
./launcher stop app
./launcher start app

I hope this helps someone.


(Jeff Atwood) #18

You only need

./launcher app rebuild

Not those three lines.

Absolutely no repro on this on a new install using Mandrill to send mail (I happen to be setting one up for a customer). Trying to log in without clicking the link in the validation mail results in the expected “not activated, click here to send another email”. Clicking the link sends another email, which arrives fine. Proof:

Not sure why you were seeing that, but I suspect mail config errors?


(Keyboard Maestro) #19

Fair enough - except I don’t understand why it managed to send the initial activation email, but then failed to send the secondary activation email if the mail settings are incorrect. Shrug.


(Jan Philip Gehrcke) #20

I also use exim4 set up on the host for sending mail. I chose a slightly different config, which I think is more systematic. First I looked up the IP of the docker bridge on the host, being 172.17.42.1 in my case. I then instructed exim to treat this as local interface and listen on it. Also, exim was told to relay mail incoming from subnet 172.17.0.0/16. The relevant keys in /etc/exim4/update-exim4.conf.conf

dc_local_interfaces='127.0.0.1 ; ::1; 172.17.42.1'
dc_relay_nets='172.17.0.0/16'

After update-exim4.conf and service exim4 restart, I tested SMTP access via Python in the container. After calling launcher ssh app, I ran python and attempted an SMTP connection right to the host via its IP address in docker’s private network:

>>> import smtplib
>>> server = smtplib.SMTP('172.17.42.1')
>>> server.set_debuglevel(1)
>>> server.sendmail("rofl@gehrcke.de", "jgehrcke@googlemail.com", "test")
send: 'ehlo [172.17.0.54]\r\n'
reply: '250-localhost Hello [172.17.0.54] [172.17.0.54]\r\n'
reply: '250-SIZE 52428800\r\n'
reply: '250-8BITMIME\r\n'
reply: '250-PIPELINING\r\n'
reply: '250 HELP\r\n'
reply: retcode (250); Msg: localhost Hello [172.17.0.54] [172.17.0.54]
SIZE 52428800
8BITMIME
PIPELINING
HELP
send: 'mail FROM:<rofl@gehrcke.de> size=4\r\n'
reply: '250 OK\r\n'
reply: retcode (250); Msg: OK
send: 'rcpt TO:<jgehrcke@googlemail.com>\r\n'
reply: '250 Accepted\r\n'
reply: retcode (250); Msg: Accepted
send: 'data\r\n'
reply: '354 Enter message, ending with "." on a line by itself\r\n'
reply: retcode (354); Msg: Enter message, ending with "." on a line by itself
data: (354, 'Enter message, ending with "." on a line by itself')
send: 'test\r\n.\r\n'
reply: '250 OK id=1X9bpF-0000st-Od\r\n'
reply: retcode (250); Msg: OK id=1X9bpF-0000st-Od
data: (250, 'OK id=1X9bpF-0000st-Od')
{}

This test shows that the exim4 server running on the host is reachable via SMTP from within the Discourse docker instance just fine.

You retrieve socket.error: [Errno 111] Connection refused or smtplib.SMTPRecipientsRefused: {'jgehrcke@googlemail.com': (550, 'relay not permitted')} in case the setup is incorrect.


Mandrill is going to paid plans on MailChimp