Add an offline page to display when Discourse is rebuilding or starting up

Is there an obvious reason why I can’t upload a .png >500kb on my forum after following this guide?

I see some talk about this line client_max_body_size 0; but that shouldn’t be the issue right?

EDIT: I quickly figured it out after posting here. Needed to check :white_check_mark: force https in the settings. I will leave this post here in case future people have an issue

3 Likes

I note that here:

nginx recommend disabling the Connection: close header as well as setting proxy_http_version 1.1 — so something like this:

    proxy_http_version 1.1;
    # Disable default "Connection: close"
    proxy_set_header "Connection" "";

I can’t find any documentation on whether Connection: close has any impact on unix domain sockets, but since this documentation is also useful for running an external proxy on a separate system and I’d expect that removing the header won’t hurt, it might make sense to recommend removing it here?

1 Like

If you have deployed this on a system with SELinux (enforcing), you can’t use the unix domain socket for the host to talk to the container, because even if you relabel the unix domain socket, it will be recreated without the label every time you restart the container. Instead, you’ll have to make two changes.

You will need to allow nginx to access the error pages, and switch from proxying over a unix domain socket to a port. That will cost a couple µs latency per request as the prices of running nginx with SELinux as one layer of security, but this won’t be perceptible to your users.

First, run these commands to allow nginx to make network connections and access error pages:

setsebool -P httpd_can_network_connect 1
semanage fcontext -a -t httpd_sys_content_t /var/www
restorecon -R -v /var/www

Then in your app.yaml, comment out or remove the - "templates/web.socketed.template.yml", expose port 80 as a different port on the local machine and rebuild the container.

expose:
  - "8008:80"   # http

Don’t use https here — you have terminated SSL in the external nginx, and the X-Forwarded-Proto header tells Discourse that the request came in via https. Make sure that port 8008 (or whatever other port you have chosen) is not exposed publicly by your firewall settings.

Then modify your external nginx configuration from proxying via nginx.http.sock to http://127.0.0.1:8008 (or your chosen port) and clear the default Connection: close header, so that the external nginx doesn’t have to establish a new IP connection for every request.

...
  location / {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    # Disable default "Connection: close"
    proxy_set_header "Connection" "";
...
1 Like

Hey @sam (and maybe @falco). I’m tasked with cleaning up some of these #documentation:sysadmin docs. This one has very high reads and I think is one of the least helpfu.

Do you think it makes sense to write a replacement that spins up Docker Hub and Docker Hub, perhaps with docker compose, making the nginx container mount the certs from the discourse container and have haproxy in tcp mode to do something like this (I’m sure this won’t work, but I presume I can figure out what will):

backend my_app_be
	balance roundrobin
	option httpchk HEAD /srv/status
        server discourse app:443 check
	server fallback nginx:80 check backup

I think that might be a workable solution that’s easier to follow than this topic. I’d then leave this one for historical purposes (and perhaps close it?) but link to the one described above.

3 Likes

Note that this topic has significant overlap with that one:

That topic has received significant updating lately (that increased the overlap even more). Maybe it makes sense to merge the offline-page part from here to there as a note (since it’s easy to add if you already run a separate Nginx instance), then mark this one as deprecated (linking to the alternatives)?

Your suggested HAProxy topic would still make sense in addition, as the default way to go for people not wanting to install a front Nginx for other reasons.

2 Likes

How was I supposed to know?

Oh.

But seriously, I like your solution better than mine!

And that topic says in big letters that it’s an advanced topic.

But maybe isn’t that necessary either, as most people understand nginx better anyway. I’ve started thinking about it now, though, so the hard part will be to get me to stop. :slight_smile:

3 Likes

It is always good if out there is plenty of alternativies. But this one is one of the most easier ones (quite many ones…), and familiar for quite many.

So please, don’t touch this one.

And leaving this as it is has another point too: search results. Because of hight trafic (and very limited use of tags…) trying to find anything specific here is quite hard nowadays. But this is quite easy to find and has very targeted purpose. If this topic will move to another one finding it will be more harder.

There is a reson why this is so popular… not so many aren’t that inspired to use docker or haproxy.

2 Likes

Sigh. Well, I guess that’s true too, but it’s at least 4 years out of date. I haven’t done it lately, but you don’t need to modify the files by hand anymore as acme (or something like it?) will do it for you.

What I really think is that it makes much more sense to use a two-container installation, which has little down time rather than to jump through these hoops to put a page up while you rebuild, but I can’t convince people of that either.

So maybe the thing to do is to rewrite this for how things work today.

1 Like

Aaaand it is much more difficult. I reckon it is easier to fix instructions how to install and use certbot that start teaching how, when and where update sql-side etc. containers.

Plus there is another point against docker (even Discourse works that way…): we can find a lot totally basic level questions like how to use docker at the first place. Or how to avoid typos in yml-files :wink:

And yet it works (except SSL-section is a bit confusing, but it was off even 4 years ago :wink: )

No. I’m not against another solutions. I’m very anti when old links and texts shall be moved to new locations without very solid reasons.

2 Likes

We’ll have to agree to disagree there. But I believe that there are likely lots of people who agree with you. (Maybe this solution is a “set it and forget it” solution, and the two-container solution does require paying attention to when there is a Postgres upgrade, which happens about every 2 years.)

OK. That part we agree on! So I think the way forward is to see what I can do to clean up that bit, and put the haproxy solution on hold.

1 Like

I’d love to see this done as part of Discourse but thank you @fefrei for this! Amazing work! I’ll be using Apache to do it but at least the base steps should be the same.

1 Like

OK only took me 2 hours of fiddling to make it how I want!

Discourse Maintenance Page with Apache2

As root

cd /var/discourse
nano containers/app.yml

Comment out these lines:

  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

expose:
  #- "80:80"   # http
  #- "443:443" # https

Add at the END of templates section add (must be last):

  - "templates/web.socketed.template.yml"

Note: This will make Discourse listen only on internal IP and apache2 will take over the 80/443 ports and SSL termination

Note: Discourse must be rebuilt for this to take effect:

cd /var/discourse
./launcher rebuild app

Install apache2 and certbot

apt install -y apache2 certbot python3-certbot-apache

Make a directory for the html page:

mkdir /var/www/discourse_maintenance

HTML page:
/var/www/discourse_maintenance/discourse_maintenance.html

<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="refresh" content="5">
        <title>Discourse Maintenance</title>
        <style>
            .center {
                display: flex;
                justify-content: center;
            }
            .container {
                max-width: 500px;
                padding: 50px 50px 30px 50px;
            }
            .title {
                padding-top: 20px;
            }
            h1, p {
                font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
            }
        </style>
    </head>
    <body>
        <div class="center">
            <div class="container">
                <h1 class="title">Discourse Maintenance&hellip;</h1>
                <p>We are currently upgrading the site, or performing scheduled maintenance.</p>
                <p>You'll automatically be redirected to the site once it's available.</p>
            </div>
        </div>
    </body>
</html>

Enable Proxy Module:

a2enmod proxy
a2enmod proxy_http
a2enmod headers

Apache vhost file:

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName your.discourse.domain
  ServerAdmin your@email.com
  DocumentRoot /var/www/discourse_maintenance

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  # Maintenance Mode
  RewriteEngine On
  RewriteCond /var/www/under_maintenance -f
  # safety check to prevent redirect loops 
  RewriteCond %{REQUEST_URI} !/discourse_maintenance.html$
  # redirect internally all requests to maintenance.html 
  RewriteRule ^.*$ /var/www/discourse_maintenance/discourse_maintenance.html

  ProxyPass / unix:///var/discourse/shared/standalone/nginx.http.sock|http://127.0.0.1/
  ProxyPassReverse / unix:///var/discourse/shared/standalone/nginx.http.sock|http://127.0.0.1/

  SSLCertificateFile /etc/letsencrypt/live/your.discourse.domain/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/your.discourse.domain/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

To enable maintenance run touch /var/www/under_maintenance

To disable maintenance run touch /var/www/under_maintenance

Credits: Add an offline page to display when Discourse is rebuilding or starting up for the intial idea, html page (trimmed/edited to my liking) and nginx config from which I based the Apache config on.

Edit: Suggestions welcome to make it automatic when 502/503 is the response. I tried but couldn’t get it to work like I wanted so I went with a known method that I use on other webservers when the backend application is down for maintenance etc.

2 Likes

On system reboot, this will delay the error/maintenance page until docker has spun up, which takes rather longer than booting the system. It also doesn’t give the option of system-provided SELinux protections for the system nginx. Using system nginx can, at least on a systemd-managed system with fast boot, get you the error page within a few seconds of boot. For me, this means that my systems answer with the maintenance page very quickly during system updates requiring reboot. (I’m running AlmaLinux 9 on the host, and it boots to nginx very fast.)

It might make sense to document an haproxy alternative and compare experiences, but haproxy-in-docker is not an apples-apples replacement for external nginx, and closing this topic would be a mistake.

It’s not just availability.

Using docker for external traffic via IPv4 hides external IPv6 addresses from the internal nginx and Discourse. You’ll have the same problem with haproxy. Look at your logs for 127.0.0.1 or 172.* RFC1918 local-only IP space addresses. Not using an external proxy means that all IPv6 traffic shows up as the same IP, which breaks the internal nginx zone rate limiting, considering all IPv6 traffic to be a single zone.

IPv6 matters more and more.

2 Likes

I discovered by accident this morning that this step not only avoids applying unix socket, but also removes use of the real_ip module, so that rate limiting is applied based on all connections together, rather than all connections per IP. I should probably contribute a new template with variables, but right now I’ve just added this to my app container YAML file:

run:
  - replace:
     filename: "/etc/nginx/conf.d/discourse.conf"
     from: /listen 80;/
     to: |
       listen unix:/shared/nginx.http.sock;
       set_real_ip_from 172.0.0.0/24;
  - replace:
     filename: "/etc/nginx/conf.d/discourse.conf"
     from: /listen 443 ssl http2;/
     to: |
       listen unix:/shared/nginx.https.sock ssl http2;
       set_real_ip_from 172.0.0.0/24;

I don’t know whether it makes sense to have, say, a templates/web.httpratelimit.yml file with something like that with a variable for the address but not using unix domain sockets. Thoughts on that?

2 Likes
server {
  listen 80; listen [::]:80; listen 443 ssl http2; listen [::]:443 ssl http2;
  server_name DOMAIN;
  ssl_certificate      /etc/letsencrypt/live/DOMAIN/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/DOMAIN/privkey.pem;

ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
  ssl_protocols TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  add_header Strict-Transport-Security "max-age=63072000;";
  ssl_stapling on;
  ssl_stapling_verify on;

  client_max_body_size 0;

  location / {
    error_page 502 =502 /errorpages/offline.html;
    proxy_intercept_errors on;

    proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
  }

  location /errorpages/ {
    alias /var/www/errorpages/;
  }
}

! # change your domain and path of error file

i got this script with ssl and work

2 Likes

I just tried using this guide and I couldn’t get it working.

I started with an Nginx proxy to run two sites out of a single Discourse container. I just wanted to add in the error page bit, so I skipped the parts that seemed to overlap with Run other websites on the same machine as Discourse. I must have missed a key step though. In the end I got what I needed from this DigitalOcean tutorial. It’s not hard to manually set this up, but it does seem like there must be a better way.

Given Docker as the standard way to run Discourse, this sounds better. I assume it would be the sort of thing where you’d set it up and forget about it.

3 Likes