Discourse behind reverse proxy and https

Installation
1

I’m trying to setup Discourse behind my Apache reverse proxy but I can’t get it working properly with https.

I’ve had a lot of problems getting this far. Right now I have Discourse on a server and an Apache server infront of it acting as a reverse proxy. I had a lot of problems getting it to run behind a reverse proxy in the first place since Discourse always wanted to redirect to the hostname set in app.yaml.

Somehow I got it working now though but I get Mixed-content warnings in my browser.
I have a redirect in Apache from http to https so that is working fine. But Discourse is still serving some stuff over http and I can’t seem to figure out how to force it to change it to https.

For instance the favicon is served over http and I can’t figure out how to change this.

Can I make Discourse change all the links to https without making Discourse handle the https traffic?

I tried to set:

Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"

In Apache but it doesn’t seem to help.

Checking the force https flag in Discourse doesn’t help either, it will just break the site since it will just ignore everything over http.
What should I do to get rid of the mixed content?

2

Apache2 will give you a lot of issues. Consider switching to nginx or caddy or traefik or haproxy

2 Likes
3

I got Apache2 working “no problem” in a test bed with Apache2 as a reverse proxy to a unix socket in the container:

The only difference I found (note: only a few hours of testing, nothing complete) was:

  • Apache2 will not work with with a symlink to the unix socket in the shared volume in the container;
  • Apache2 was a bit slower in a rough test, but not by much.

Personally, I’m not a fan of religious wars over technologies; so I disagree that “Apache2 will give you a lot of issues.” I did not experience any negative issues with Apache2 during my tests.

Here is the core setup I used with Apache2 (HTTP, worked fine with LETSENCRYPT, BTW):

# cat discourse.example.conf
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName  discourse.example.com
  DocumentRoot /website/discourse

  RewriteEngine On
  ProxyPreserveHost On
  ProxyRequests Off
  ProxyPass / unix:/var/discourse/shared/socket-only/nginx.http.sock|http://localhost/
  ProxyPassReverse  / unix:/var/discourse/shared/socket-only/nginx.http.sock|http://localhost/
  ErrorLog /var/log/apache2/discourse.error.log
  LogLevel warn
  CustomLog /var/log/apache2/discourse.access.log combined

  RewriteCond %{SERVER_NAME} =discourse.example.com
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

Note: The only time we experienced issues with HTTP being served even when force_https set was when files were missing in the /uploads directory, but this (of course) is not related to Apache2 v. nginx as a reverse proxy.

1 Like
4

Thanks for the reply’s but I don’t have Apache on the same server as Discourse. I might not have been to clear about that.
I have an existing Apache server with a bunch of websites and I need it to reverse proxy Discourse located on a different server so I can’t use sockets.

5

Cheers.

I have not tried this method, but you might consider mounting your remote file system and see if you can access a unix socket that way, especially if the servers are in the same datacenter and network performance across a wide-area-network is not an issue.

Without posting any architecture, operating system details, network configuration, etc, it’s hard to reply and perhaps even out of scope here at meta discourse.

Be creative!

1 Like
6

The last time I tried using that config with apache2, I was getting connection failures to the discourse message bus. It was over an year ago at this point tho. It appears as if the side is loading fine but in the F12 developer console it clearly stated that wss connections timed out after a few initial tries.

1 Like
7

You can clearly see in the example configuration I posted, we did not use wss

wss !== apache2 reverse proxy (it’s only one way to do it, and we don’t use wss)

In fact, we only use ngnix and apache2 reverse proxy configurations using unix domain sockets because:

  • I am lazy and like simple, easy to debug configurations.
  • unix domain sockets are simple and easy to debug
  • In nginx, we can switch between the reverse proxy and any container with a symbol link
  • apache2 (reverse proxy to container) does not work with a symbolic link so a web server restart is required

However, @Grunskin asked about something which we have not configured yet; doing the reverse proxy on one host and running the container on another host.

When I have time, I’ll test this for both nginx and apache2 in the same data center and see if I can get this to work mounting the remote file system and using a unix socket.

Until then…

Note: IMO, this issue is not germane to either nginx or apache2 which only acts as reverse proxies (but as mentioned, have not yet tested the remote access config, so cannot comment further.).

1 Like
8

Why is this necessary?

Discourse is an application, not a website. Once the initial javascript payload has been delivered to your browser many of the features rely on a snappy connection to the discourse server. Proxying via another system is going to introduce latency and seriously degrade user experience.

Can you explain the reasoning behind your need?

2 Likes
9

There are plenty of reasons for using a reverse proxy.
For example if I only have 1 public IP and need multiple webservers to be public on port 80/443, and I can’t run Discourse for this example on that particular webserver.
Use it as SSL Offloading so the end-server doesn’t have to deal with encryption.
The attack surface on the end-server is reduced by placing it behind a reverse proxy.
There are lots of legitimate reasons for this and I can’t seem to understand why this wouldn’t be possible.

After a while I remembered that I already have another server with Discourse running behind my Apache server and it has been working fine for at least 2 years. I’ve configured the new Discourse the same way but can’t get it to stop serving some stuff over http. The only difference between the Discourse servers are that the old one is running 2.4 and the new one 2.5 so I don’t know if there is any differences there?

As I said in the first post force_https breaks the site, making it impossible to sign in, accept invites etc. It seems like there are some javascripts that won’t run since they’re probably served with http.
Wouldn’t it make more sense to make force_https rewrite all http links to https rather than discarding them? At least have it as an option

What is the recommended way of setting up Discourse publicly? Setup a server in a DMZ with it’s own external IP?

10

I suggest to take a look to Traefik.
It works great and manages SSL certificates automatically.

11

There we plenty of reasons to run behind a reverse proxy. I’m pretty sure that Discourse.org’s infrastructure runs behind HAproxy.

I use treafik and vcaddyserver, and have made nginx, HAproxy, and even apache work in the past (for a subfolder install query wordpress).

This is your problem. You need to enable b force_https and figure out why it’s breaking. Turning it off isn’t an option. You asked for free support and those who responded don’t have a solution for apache, so you’ll have to be the leader of the apache band.

2 Likes
12

Yes. We totally agree.

We now use the “two container with reverse proxy” on all of our sites, production, testing and staging, except for the server we are using for staging our main migration.

This is because it is easier to run all the various migration scripts when we have postgres, mysql, the discourse app code all in a single container. This is non-production and easier to debug. But when we are happy, we move move the backup to production and restore.

One issue with this is that even the super-duper two container setup with reverse proxy cannot compensate for the downtime during a DB restore, since there is only one DB.

Maybe in the future, we will try a setup with two DB containers so we can restore one and switch back and forth… on the data side as well.

Or better yet, we will restore to a database with a different name and switch in real time, but we don’t know how to do this yet. If you know where in the code we can change the name of the production DB from discourse to discourse2 without rebuilding the entire app, that would be very good :slight_smile: Maybe creative and innovative @pfaffman super consultant knows?

Update: It’s here in templates/postgres.template.yml :slight_smile:

(I see some interesting mv postgres folder stuff in here for sure :slight_smile: :). )

Both standalone and multi-container have advantages and disadvantages; but for production we are fully into the two container with reverse proxy configuration. Migration testing and staging, standalone is best for us.

Regarding Apache2, I with I had time to set up and test this with the reverse proxy on one server and the containers on another server. Sorry about that…

Also, I need to come up with a way to keep the site up when in the two container config and doing a DB restore. I see (now) the templates/postgres.template.yml template is geared toward a single standalone container config.

1 Like
14

Just want to add that Discourse doesn’t use WSS. Message Bus uses Long Polling with chunked encoding (streaming).

4 Likes
15

You have to set that hostname to the hostname that Discourse is going to be accessed by. If the domain name in app.yml isn’t the one that people type in their browser to get to your site it won’t work.

You don’t have the ssl or letsencrypt templates in your app.yml, do you?

You do need force_https turned on.

And you have to jump through some kind of hoops to get long polling to work. I don’t remember what they are.

1 Like
16

Hi @Grunskin

We understand your frustration. However, when you set:

force_https = true

This is not the problem. As @pfaffman mentioned above (at least twice, one of meta’s top migration experts):

You “must” leave force_https = true and then figure out the “real problem”.

If I were you, based on what. I have read.

First, I would set up your reverse proxy on the same server as the discourse container(s) to simplify your problem, only for testing and troubleshooting. Make sure this simple test case works perfectly. Then, when it works on the same host, turn off that test reverse proxy, and move to your desired dual server setup.

This is an interesting problem. You can solve this if you leave force_https = true and maintain a structured, step-by-step troubleshooting method. The kinds of problems are just IT puzzles begging to be solved.

You can do it.

PS: If you get discouraged or bored with this puzzle, you can always throw some cash as @pfaffman or some other experienced meta person and pay them to get you past this roadblock and on to bluer skies ahead.

Note: We had zero issues getting Apache2 to work as a reverse proxy with a unix domain socket on the same server. My sincere apologies that I don’t have the personal time to set up the two server configuration and get the Apache2 reverse proxy method to work on two different servers. We are busy with final migration tasks cleaning up “crazy bbcode abuse” to markdown issues and it’s taking more time than we thought, originally.

1 Like
17

I decided to go the route of putting on discourse in localhost on port 8443 in SSL, lock 8443 with UFW (firewall) and proxy 443 to 8443 with apache2. Trying now.

                ProxyPass / "https://localhost:8443"
                ProxyPassReverse  / "https://localhost:8443"
1 Like
18

I migrated our discourse to a new setup and now have problems with http 403 on POST session during the login. I would guess a CSRF problem, but right now I have no idea where to start the debugging and /var/log/discourse-var-log/production.log and production_error.log all have 0 bytes …

Any ideas how to debug this properly?

The current setup:
1. haproxy as central https load balancer/accelerator (for both discourse and other services)

forum >> develd apache rev. proxy p.82
backend forum-backend
   mode http
   server forum.netzwissen.de 10.10.10.14:83 cookie A check
   http-request set-header X-Forwarded-Port %[dst_port]
   http-request add-header X-Forwarded-Proto https if { ssl_fc }
   # HSTS header, 16000000 seconds: a bit more than 6 months
   http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"

2. local apache as rev proxy

   <IfModule proxy_module>
    ## <https://meta.discourse.org/t/running-other-websites-on-the-same-machine-as-discourse/17247>
    ProxyPreserveHost On
    # ProxyRequests Off     
    RequestHeader set X-Forwarded-Proto expr=%{REQUEST_SCHEME}
    RequestHeader set X-Real-IP expr=%{REMOTE_ADDR}
    ProxyPass /  unix:/var/discourse/shared/web-only/apache.http.sock|http://localhost/
    ProxyPassReverse  / unix:/var/discourse/shared/web-only/apache.http.sock|http://localhost/
    </IfModule>

3) Discourse operating with separate web_only and data containers, web_only deployed with - "templates/web.socketed.template.yml"

19

This is the session request upon login that fails:

POST /session HTTP/1.1
Host: forum.netzwissen.de
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: */*
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 89
X-CSRF-Token: dtV0N6faVQSWZsg6z9ZGOxQBjuTpBZk6tAMRxaXJdwozF1kObw9UuiFnxbLf5OGDeL1DWDgZ5W3oJP7CY+LwRw==
Discourse-Present: true
X-Requested-With: XMLHttpRequest
Origin: https://forum.netzwissen.de
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Referer: https://forum.netzwissen.de/
Connection: keep-alive

Reply from the web_only containers webserver:

HTTP/1.1 403 Forbidden
date: Sun, 13 Mar 2022 16:41:56 GMT
server: nginx
content-type: text/plain; charset=utf-8
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-download-options: noopen
x-permitted-cross-domain-policies: none
referrer-policy: strict-origin-when-cross-origin
vary: Accept
x-request-id: 778da942-3c1c-493b-946b-478984f53a8c
x-runtime: 0.003623
transfer-encoding: chunked
strict-transport-security: max-age=16000000; includeSubDomains; preload;
20

I am not familiar with the csrf stuff and also not with nginx (the external webserver is an apache 2.4) , but I am pretty shure that CSRF is my problem here as the discourse works fine without logon and only the POST requests which are used for login fail here. My central haproxy has the internal IP 10.10.10.21 , therefore I’ve put

set_real_ip_from 10.10.10.21/24;

in the yml for the discourse.conf in the web_only container. I also tried with the default 127.0.0.1/24; but both result in the same 403 error upon login.