Using a certificate when Discourse is installed behind a reverse proxy

So I’m having a lot of problems trying to setup SSL with Let’s Encrypt.

Previously I started this topic, as I’m setting up Discourse on an Ubuntu VM. Going to localhost in a broswer on my Ubuntu VM with Discourse installed does bring up the “Congratulations!” (when I try to NOT set up SSL in app.yml) … but even still, it will NOT send a confirmation email when I try to register the Admin account, but I’m not going to worry about that issue another time and instead focus on an issue I’d like to solve beforehand:

I seem to be having a lot of trouble with generating a Let’s Encrypt Key/Cert pair. I have gone through the first 4 steps in the Let’s Encrypt Guide, but I run into errors: when I run ./launcher logs app, I receive the following:
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
supervisor pid: 53 unicorn pid: 79

I have also found that under /var/discourse/shared/standalone/ssl I found the CER and KEY file, but the CER file is 0 bytes in size. Likely something went wrong here. There is also no PEM file as per the example in the Let’s Encrypt guide.

I also tried manually reissuing the cert as per the Guide, but when I got to /usr/sbin/nginx -c /etc/nginx/letsencrypt.conf well … it reports that there is no letsencrypt.conf (which, I checked, and that file is not in /etc/nginx).

Anyone here know what’s going on? Would you recommend that the Admin Activation Email needs to be able to send first BEFORE I try to set up Let’s Encrypt?

Is this an internet-connected server, or a VM on a local PC?

Did you take note of this

@Stephen It’s a VM hosted on ProxMox VE 5.3. The OS on the VM that I installed Discourse on is Ubuntu Server 18.04 LTS.

ProxMox, and the VM itself, are both connected to the Internet. No problems there. Both have Static IPs properly assigned to them through my router.

Note: ./discourse-setup will enable Let’s Encrypt. And as of March 2017, you can run it again, and press return a few times and enter your email address ; the script will include the required templates and insert your email address as required.

@pfaffman So you’re suggesting I run through ./discourse-setup again from scratch, but this time enable Let’s Encrypt through the Setup? Or am I understanding your suggestion incorrectly?

You’ve skirted around answering my question somewhat.

Is there a valid public DNS record pointed at the public IP of the server? Are :80 and :443 externally accessible on that hostname and IP?

1 Like

There indeed is. The Internal IP on the Discourse VM is 192.168.0.104. I created the DNS (let’s say its boards.myreserveddns.com) and I assigned it the local IP.

You may be asking why I assigned the DNS the Internal IP as opposed to the External IP. Well … I’ve got an Nginx VM that I need to have the External IP assigned to because I have multiple VMs set up on the Server (which require I have Nginx reverse proxy the traffic through the Internal IPs). I know I have Nginx somewhat set up correctly because I have an ownCloud VM that I have set up with HTTPS and a DNS name which is all working just fine.

Assuming you’ve configured your DNS and network correctly, the only things you need in your app.yml are:

uncomment in templates:
- "templates/web.ssl.template.yml"
- "templates/web.letsencrypt.ssl.template.yml"

uncomment in expose:
- "443:443" # https

add an email to LETSENCRYPT_ACCOUNT_EMAIL:

That’s your problem, Let’s Encrypt won’t work within Discourse if you’re using a reverse proxy in front of it. It’s not possible for the Discourse container to make the request for the certificate if it’s not directly accessible. The reverse proxy needs to request the certificate and include it in the server declaration. Discourse will continue to listen on :80 without a certificate, Let’s Encrypt will handle the external encapsulation. Look for guides here on Meta for installing Discourse alongside another webserver.

You will need to manually enable force_https on your site afterwards to get rid of mixed content errors. You will also need to make sure your reverse proxy configuration hands off the true origin IP, otherwise you won’t see user IP addresses within Discourse and will get all kinds of funky rate limit errors.

If you’re familiar with nginx this shouldn’t be a problem for you to do.

The external DNS entry needs the external IP address, even if a reverse proxy is in play.

1 Like

Yes if discourse is the only site on the machine then you can just enable let’s encrypt with discourse-setup.

2 Likes

But, he doesn’t? In fact he said the opposite:

1 Like

Already did all that, but I assume you were suggesting this before you read that I want it Reverse Proxying through Nginx.

Ok, so I need to revert app.yml for an HTTP setup? Do I still need to have a Key and Cert file pair in /var/discourse/shared/standalone/ssl? And if so, how in fact would I generate the necessary Key and Cert file pair I need on my Nginx VM?

I assume you’re suggesting that I follow one of these guide?

Any recommendations? Even skimming these guides, I’m wondering: are they assuming that Nginx will be installed on the same VM as Discourse or on a separate VM? I’ve skimmed both and its not obvious to me.

I’m unfortunately not familiar with Nginx much at all. I am only using it based on the recommendation of the previous Admin who helped set me up our current (and soon to be taken offline) Discourse Server. As per what he told me: it was on its own IP, but this new Discourse is on a VM on a single Server, so Reverse Proxying with Nginx is required.

I don’t understand what you mean when you say “manually enable force_https on your site afterwards to get rid of mixed content errors.” So what you’re suggesting is that I get Discourse working via HTTP first? But then I still have questions:

  1. How would I go about setting up HTTPS on a Discourse reverse proxying through Nginx, particularly in terms of creating Key and Cert file pairs? I guess I’m going to hope that’s covered in one of these Nginx + Discourse guide.
  2. I wouldn’t even know where to start with “manually enable force_https” as you suggested. Would you mind providing more details?

Weird, for setting up my ownCloud VM DNS entry (which Reverse Proxies through Nginx), I used the local IP and it connects just fine. I guess for Discourse I will make sure it uses my External IP.

Ok, made some more progress. Been following THIS guide mostly, although I’ve followed THIS guide’s advice of not including the ssl_certificate nor the ssl_certificate_key in the Nginx CONF file, as nginx -t was reporting errors.

nginx -t on the Nginx Server nor Discourse Rebuild are reporting errors right now, however localhost will not bring up the “Congratulations” message, but https://boards.myreserveddns.com:2045/ will bring up a “Secure Connection Failed” in the VM’s local browser (SSL received a record that exceeded the maximum permission length) and I get a “Connection Timeout” outside of the VM’s local browser.

All I’ve done on my router is Port Forwarded the following:

Port IP Address Explanation
80 192.168.0.101 Nginx VM (HTTP)
443 192.168.0.101 Nginx VM (HTTPS)
2045 192.168.0.104 Discourse VM (Port exposed in place of 80)

And I have the following DNS Settings (and let’s say my Local IP is 1.2.3.4):

Host IP Address Explanation
ngx.myreserveddns.com 1.2.3.4 Nginx DNS
board.myreserveddns.com 192.168.0.104 Discourse DNS

Right now I am assuming I should also set the Discourse DNS to 1.2.3.4 … but I don’t want to touch anything on the DNS right now until I receive recommendation based on the rest of my setup below.

Here is the /etc/nginx/sites-available/discourse.conf file in the Nginx VM:

server {
    listen 192.168.0.101:80; listen [::]:80;
    server_name boards.myreserveddns.com;

    return 301 https://$host$request_uri;
}

server {
# The IP that you forwarded in your router (nginx proxy)
 listen 192.168.0.101:443 ssl http2;

# SSL config
# ssl on;
# ssl_certificate /etc/nginx/ssl/0000_csr-certbot.pem;
# ssl_certificate_key /etc/nginx/ssl/0000_key-certbot.pem;
 include /etc/nginx/snippets/ssl.conf;

# Make site accessible from http://localhost/
 server_name boards.myreserveddns.com;

# The internal IP of the VM that hosts your Apache config
 set $upstream 192.168.0.104:2045/;

 location / {

 proxy_pass http://$upstream;
 proxy_set_header Host $host;
 proxy_set_header X-Forward-Proto $scheme; #X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_http_version 1.1;
 proxy_redirect http://$upstream https://boards.myreserveddns.com/;

 }
}

And here is the /var/discourse/container/app.yml file in the Discourse VM:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache/nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "2045:80" # http

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "768MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 4

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: board.myreserveddns.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'admin@myreserveddns.com,postmaster@myreserveddns.com'
  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: smtp.sparkpostmail.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: SMTP_Injection
  DISCOURSE_SMTP_PASSWORD: "<SMTP_Password>"
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: admin@myreserveddns.com

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-chat-integration.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
# - exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
# - exec: rails r "SiteSetting.notification_email='postmaster.myreserveddns.com'"
  - exec: echo "End of custom commands"

One thing I did notice was that I have been unable to Bootstrap the app.yml file (./launcher bootstrap app). What could be the problem here?

FAILED
--------------------
Pups::ExecError: socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 failed with return #<Process::Status: pid 44 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1"
<SMTP_Password>
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

<SMTP_Password> is not literal, just keeping out of the post.

Keep in mind that ./launcher rebuild app reports no errors … so I’m stumped with the Bootstrap error.

EDIT: Ah, just read this topic. Looks like there is no need to Bootstrap if I can rebuild the project without errors. I suppose the main issue is related to the “Secure Connection Failed”, which I suspect is an issue with the Key/Cert and/or the Address Reservation settings on my Router.

1 Like

UPDATE: So when I updated my DNS so my Public IP (1.2.3.4) is pointing to boards.myreserveddns.com, here is what I get when I type various addresses in Firefox:

URL What happens?
http://boards.myreserveddns.com Forwards to https://boards.myreserveddns.com and receive “Secure Connection Failed: The connection to board.myreserveddns.com was interrupted while the page was loading.”
https://boards.myreserveddns.com (same as above)
http://boards.myreserveddns.com:2045 Congratulations, you installed Discourse!
https://boards.myreserveddns.com:2045 Receive "Secure Connection Failed: An error occurred during a connection to board.myreserveddns.com:2045. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

(That last entry comes out as “board.myreserveddns.com took too long to respond.” in Chrome)

Get these sets of messages in BOTH the local browser of the VM as well as a browser on my home PC.

I think I’m very close to figuring this out, mostly a matter of updating something in the Discourse app.yml file or the Nginx’s CONF file for Discourse perhaps? @Stephen, @pfaffman or anyone else have any ideas?

@Stephen and @pfaffman any ideas?

The Discourse configuration is complete here.

Those symptoms are constant with an inbound firewall blocking https from reaching your nginx container.

1 Like

@riking I don’t see how the Firewall would be the issue. I ran a sudo ufw status verbose on both the Discourse VM and Nginx VM and have the following (among other values):

To                         Action      From
--                         ------      ----
587                        ALLOW IN    Anywhere
2045                       ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere

Yes I even manually “allowed” ports 587 and 2045 in both UFW Firewalls for the Discourse VM and the Nginx VM.


Now, I did start up a topic about this issue over on the Let’s Encrypt forum. I’ve updated the Nginx CONF directory for the Discourse VM, and receiving an Expecting: TRUSTED CERTIFICATE error.

I suspect that is currently the issue. I’ve been told that Discourse needs to be setup for HTTP and Nginx needs to handle the HTTPS … do I need to create the Key/Cert pair on the Discourse or Nginx VM?

The cert is handled by the external nginx, not discourse.

Right, then that means I’ll need to create the Key/Cert pair on Nginx, as I thought. I’ll have to ask the Let’s Encrypt community on the best way to do this.

That’s who to ask, but you should just be able to run certbot for nginx. It’s pretty standard.

1 Like

@pfaffman I was able to run certbot. Here are the results … they haven’t changed anything that I’ve noticed, as per this post on the Let’s Encrypt forum:

I’ve run into the issue before where the “Apache Ubuntu Default Page” was what came up with a previous Discourse install on a different server … however, on that server I actually HAD installed Apache. Neither the Nginx VM nor the Discourse VM had Apache installed on them (as mentioned in the linked post).

Also mentioned there: I’m tempted to try installing Nginx straight to the Discourse VM itself and running through the Let’s Encrypt key/cert process on that server, see if that does any good.

It won’t.

Above you said that you are using a reverse proxy to run multiple sites behind a single IP. If you have more than one site on 443, then all of the certificates and SSL encapsulation has to occur on the proxy, using a SAN certificate. The SSL session has to terminate at the reverse proxy and the host header inspected so that it can be passed on unencrypted.

There really is no way around this, is there?

I really wish there was a more clear guide with setting up the Let’s Encrypt Key/Cert pair. I guess I will have to keep communicating with the Let’s Encrypt community.