Add an offline page to display when Discourse is rebuilding or starting up

:warning: This guide is intended for advanced users, who are already using nginx outside the docker container. By following this guide you make your setup more complicated and will loose some speed benefits like HTTP2 if you’re not running Ubuntu 16.04 or later. Proceed with caution!

When Discourse is rebuilding or starting up, your users will usually either see an error message from their browser…

…or a not-so-nice 502 error message from Nginx:

If you’re a perfectionist like me, you’ll probably find that unacceptable. Fortunately, fixing this is quite straightforward – so let’s dive right in!

To customize this guide, simply put the domain your Discourse is running on into the box below:

If you’re already using HTTPS and have it set up inside the container (you are using web.ssl.template.yml and possibly web.letsencrypt.ssl.template.yml), this howto will move your SSL setup out of the Docker container into Nginx running on the host, and request a new certificate from Let’s Encrypt. This is necessary because we need SSL to work even when Discourse is rebuilding its Docker container or otherwise unavailable. This will break auto-renewal, so you’ll need to manually renew every three months or set up auto-renewal on the host.

Set up nginx

If we want nicer error messages, we’ll need to set up a front-end server that usually forwards all requests to Discourse, but injects our error message when it cannot reach Discourse. This howto uses nginx as the front-end server.

:bell: If you’ve already set up nginx on your host (for example to run other websites on the same machine as Discourse), you can skip this section and continue with the next one.

To clear up ports for nginx, we must first tell Discourse to listen on a socket, not on the normal ports:

cd /var/discourse
nano containers/app.yml

Comment out the lines with web.ssl.template.yml and templates/web.letsencrypt.ssl.template.yml if they are in use (we’ll set up HTTPS on the host later on), add

  - "templates/web.socketed.template.yml"

as the last line in the templates: section, and comment out all ports from the expose: section.

Here’s how it should look:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"
  - "templates/web.socketed.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  #- "80:80"   # http
  #- "443:443" # https

When you’re done, save the file, exit and run this:

./launcher rebuild app

If all went well, Discourse will be unreachable because it is now only listening on the local socket. Don’t worry, we’ll soon fix that!

Next, install nginx:

apt-get update
apt-get install nginx

Let’s add a place to host local web content, and then edit the default Nginx default configuration file:

mkdir /var/www
nano /etc/nginx/sites-available/default

Add HTTPS

While we’re here, we’ll configure Nginx to redirect all requests to HTTPS, and to allow requesting a free certificate from letsencrypt. Replace the contents of /etc/nginx/sites-available/default with this:

server {
        listen 80; listen [::]:80;
        server_name =DOMAIN=;

        location /.well-known/acme-challenge/ {
                root /var/www;
        }

        location / {
                return 301 https://$host$request_uri;
        }
}

Apply the changes with

service nginx reload

Now we can set up Let’s Encrypt and get a certificate. If you are running Ubuntu 16.04 or later, you can use the official package:

apt-get update
apt-get install letsencrypt
letsencrypt certonly --webroot -w /var/www -d =DOMAIN=

For other operating systems you’ll need to get certbot installed manually and issue the cert like so:

mkdir /var/letsencrypt
cd /var/letsencrypt
wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto
./certbot-auto
certbot-auto certonly --webroot -w /var/www -d =DOMAIN=

Any errors? Did you get your certificate? If so, let’s proceed!

:alarm_clock: If you installed certbot from your package repository, renewals usually happen automatically. Otherwise, set a reminder to run letsencrypt renew && systemctl reload nginx.service before your certificate expires!

Let’s edit the Nginx config again to add HTTPS support:

nano /etc/nginx/sites-available/default 

The old server block in the file needs to stay – it redirects all your users to HTTPS. We need to add a new server block below the old server block:

server {
  listen 443 ssl http2;  listen [::]:443 ssl http2;
  server_name =DOMAIN=;

  # ssl on; <-- This directive is deprecated, use `listen 443 ssl` instead.
  # change these paths as necessary
  ssl_certificate      /etc/letsencrypt/live/=DOMAIN=/fullchain.pem;
  ssl_certificate_key  /etc/letsencrypt/live/=DOMAIN=/privkey.pem;

  ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
  ssl_protocols TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  add_header Strict-Transport-Security "max-age=63072000;";
  ssl_stapling on;
  ssl_stapling_verify on;

  client_max_body_size 0;

  location / {
    proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

If you’re using a version of nginx that cannot speak HTTP2 yet, remove http2 above (twice). The version of nginx offered in Ubuntu 16.04.1 can handle HTTP2.

:warning: Remember to modify the site name and paths as needed above. If you modified the volumes: section of your app.yml, you’ll also need to change the proxy_pass line.

Let’s reload the config we just changed:

service nginx reload

Your site should now be back up, securely over HTTPS. :tada:

Now tell Discourse that it should always use HTTPS URLs by checking force https in Site Settings.

:bell: We recommend that you run SSL Server Test (Powered by Qualys SSL Labs) on your site to verify that your https settings are secure and safe.

Create an error page

Next, you’ll have to design an error page to show when Discourse is offline. Let’s create a path for it.

mkdir /var/www/errorpages
  • If you’re a talented designer, feel free to build a beautiful page yourself, and share it here!
  • If you need to use external resources like images, load them from /errorpages/.
  • I recommend that you include <meta http-equiv="refresh" content="120"> in your page – this will refresh the page every 120 seconds, which means that Discourse will load automatically once it’s available again.
  • Name your main HTML file discourse_offline.html, and place all files in /var/www/errorpages/.

If you’re happy with a page made by an untalented designer, you can simply steal my design instead :blush:

discourse_offline.html (1.9 KB)
d-logo-sketch.png (14 KB)
sob.png (1 KB)

(If you need help copying these files to your server, this post may help.)

Once you’re done, edit the Nginx config to serve the page you just created:

nano /etc/nginx/sites-available/default 

Simply add

location /errorpages/ {
    alias /var/www/errorpages/;
}

to the HTTPS server section of your nginx config, and then reload.

service nginx reload

Test by visiting https://=DOMAIN=/errorpages/discourse_offline.html in your browser – you should see your new error page :no_entry:

Serve your error page

Finally, we’ll set up nginx to serve your error page when it cannot reach Discourse, or when Discourse isn’t ready yet. Add the following lines to the location / block:

error_page 502 =502 /errorpages/discourse_offline.html;
proxy_intercept_errors on;

Apply your changes, as usual:

service nginx reload

If you want to test your settings, run

cd /var/discourse
./launcher stop app

to take your site offline and try to visit it:

Don’t forget to run

cd /var/discourse
./launcher start app

afterwards so your Discourse is available again!

P.S: Don’t hope or try to repeat this procedure more than 2-3 times in a week. Because Letsencrypt won’t allow you more than 5 times to fetch certificates from their servers. If you struck that limit anyway, either you’ll have to wait till 7 days, counting from the day you requested 1st certificates in the series of 7 tries, or you’ll have to setup your site in some sub-domain, like by pre-pending ‘www’ to it.

97 Likes

To make the cert auto renew in Ubuntu 14.04 LTS, I did this:

cd /etc/cron.daily
nano letsencrypt-renew

and added to the file this renew command, taken from the certbot docs:

#!/bin/bash
/var/letsencrypt/certbot-auto renew --quiet --no-self-upgrade

I then had to change permissions on the file so it could run:

chmod 755 letsencrypt-renew

After that I was able to run the bash script manually and it seemed to work! It’s only daily, the docs recommend twice a day, but once a day seems like enough renewal opportunities to me, since the certs expire every 90 days.

8 Likes

One big issue here is that you will be killing http2 support on latest chrome unless you a rerunning absolute latest Ubuntu, which you are not.

4 Likes

Why not put the reverse proxy (nginx with the latest openssl and let’s encrypt) in another Docker container? :wink:

1 Like

Personally if I wanted the best experience

  • I would add an haproxy in tcp mode container to farm reqs to either online/offline container

  • have the nginx offline container mount the same SSL volume so you don’t need to fuss with running lets encrypt in cron and simply have existing template run it

  • have a data container so I can bootstrap while online

nginx proxying to nginx is tricky to get right, for example the setup here is restricting uploads to 2 megs, but plenty of other caveats

3 Likes

Why is that? I have a super similar setup running in production, and just successfully uploaded a 10 MB file to it.

Correction, 20mb :slight_smile: but really you want to set that to 0

5 Likes

Good suggestion, I’ve done that.
(I also made the post a wiki post.)

Honestly, I would be significantly less worried about this stuff if the offline/proxy app was running in a container, so much less error prone

I worry that there is way too many manual steps here and underlying versions are unknown

3 Likes

You may be right – but to my defense: We’ve been recommending a setup like this for running other sites on the same host for two years now.

In my defence that howto is using “listen spdy” which kind of proves my point :slight_smile:

https://github.com/discourse/discourse_docker/blob/master/templates/web.ssl.template.yml#L21

Cc @riking

3 Likes

So it looks like @sam is pretty strongly opposed to the current approach, and looking at how difficult it is to get a newly compiled Nginx on Ubuntu 14.04 with OpenSSL 1.0.2 support (absolutely required for http/2 to work), I tend to agree. @mpalmer says Ubuntu 14 will never get a new version of OpenSSL in any form via apt-get, and 1.0.1 is end of life in December of this year.

(Is it possible to copy the nginx binary from Ubuntu 16 to Ubuntu 14? Does that work? That might be simplest, if it does…)

Anyway, it looks like we need another container here, a container that holds Nginx to do the routing and so on.

6 Likes

Doesn’t look like it; the package dependency for, ironically, libssl1.0.0 is the sticking point. It might work, if the ABI extensions in libssl1.0.0 aren’t relied upon, but it’d be a risky thing to try in general. I recommend a separate nginx-only container.

1 Like

This change seems to cause problems with https redirects for logins, too – even with use https (force https) checked. After instituting the nginx outer layer with https, I had to:

  • enable http return URL for Google logins
  • enable http return URL for GitHub logins

Otherwise you get errors, e.g.

(github) Authentication failure! redirect_uri_mismatch: OmniAuth::Strategies::OAuth2::CallbackError, redirect_uri_mismatch | The redirect_uri MUST match the registered callback URL for this application. | https://developer.github.com/v3/oauth/#redirect-uri-mismatch

These are problems I didn’t have when the site was using Let’s Encrypt inside the container…

3 Likes

I tried to research that and think this is a Discourse bug.

Enabling use https should cause Discourse to only use HTTPS URLs, but neither the Github authenticator, the Google authenticator nor their superclass inspect this site setting. Omniauth tries to detect SSL like this:

https://github.com/intridea/omniauth/blob/1cc1cf4b2821a7d2a4a376a5ca93c61b6bd8b5f1/lib/omniauth/strategy.rb#L499

Adding

proxy_set_header X-Forwarded-Proto $scheme;

to the nginx configuration might work around that, but I’m not sure this is passed on by Discourse’s internal nginx and cannot test that right now.

I do know that SSO is not affected.

5 Likes

@fefrei I am not sure this is needed, because I just rebuilt talk.commonmark.org using the standard web updater. I had two browser windows open:

  1. Home page of site in one window, in anon mode
  2. /admin/upgrade rebuild process going in one window

I mashed f5 like an insane :monkey_face: in window #1 and at no point during the rebuild was the site unavailable to me as an anonymous user hitting the homepage. Screenshot proof:

So… if there is never an outage during a typical web update… is this complex hack really needed?

1 Like

This is in place to serve a message during rebuilding and the following boot-up – web upgrades should always be fine :slight_smile:

(It may not be worth it for a stable stock site. I think it’s definitely worth it if you either already have to use a nginx reverse proxy for another reason, or if you rebuild often, e.g. to install or uninstall plugins.)

1 Like

There is some stuff I don’t like about this

  • it is quite complex
  • it screws up http/2 completely (unless you are on Ubuntu 16)
  • it does not help in the typical update case only in the rare full rebuild case
1 Like

I agree, as long as this is the only reason for a front-end nginx instance.

If you need to have that for some other reason, you can start with the Create an error page section – the rest is really straightforward. (And if you need to set up a front-end nginx, I think these steps are easier to follow than these instructions.)

3 Likes

Should I delete the rest of the template already included in /etc/nginx/sites-available/default

2 Likes