Adding an offline page when rebuilding

(Felix Freiberger) #1

:warning: This guide is intended for advanced users, who are already using nginx outside the docker container. By following this guide you make your setup more complicated and will loose some speed benefits like HTTP2 if you’re not running Ubuntu 16.04 or later. Proceed with caution!

When Discourse is rebuilding or starting up, your users will usually either see an error message from their browser…

…or a not-so-nice 502 error message from Nginx:

If you’re a perfectionist like me, you’ll probably find that unacceptable. Fortunately, fixing this is quite straightforward – so let’s dive right in!

In this howto, I’ll use as the domain Discourse is running on – simply replace it with your domain whenever you see it.

If you’re already using HTTPS and have it set up inside the container (you are using web.ssl.template.yml and possibly web.letsencrypt.ssl.template.yml), this howto will move your SSL setup out of the Docker container into Nginx running on the host, and request a new certificate from Let’s Encrypt. This is necessary because we need SSL to work even when Discourse is rebuilding its Docker container or otherwise unavailable. This will break auto-renewal, so you’ll need to manually renew every three months or set up auto-renewal on the host.

Set up nginx

If we want nicer error messages, we’ll need to set up a front-end server that usually forwards all requests to Discourse, but injects our error message when it cannot reach Discourse. This howto uses nginx as the front-end server.

:bell: If you’ve already set up nginx on your host (for example to run other websites on the same machine as Discourse), you can skip this section and continue with the next one.

To clear up ports for nginx, we must first tell Discourse to listen on a socket, not on the normal ports:

cd /var/discourse
nano containers/app.yml

Comment out the lines with web.ssl.template.yml and templates/web.letsencrypt.ssl.template.yml if they are in use (we’ll set up HTTPS on the host later on), add

  - "templates/web.socketed.template.yml"

as the last line in the templates: section, and comment out all ports from the expose: section.

Here’s how it should look:

  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"
  - "templates/web.socketed.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see for details
  #- "80:80"   # http
  #- "443:443" # https

When you’re done, save the file, exit and run this:

./launcher rebuild app

If all went well, Discourse will be unreachable because it is now only listening on the local socket. Don’t worry, we’ll soon fix that!

Next, install nginx:

apt-get update
apt-get install nginx

Let’s add a place to host local web content, and then edit the default Nginx default configuration file:

mkdir /var/www
nano /etc/nginx/sites-available/default


While we’re here, we’ll configure Nginx to redirect all requests to HTTPS, and to allow requesting a free certificate from letsencrypt. Replace the contents of /etc/nginx/sites-available/default with this:

server {
        listen 80; listen [::]:80;
        server_name;  # <-- change this

        location /.well-known/acme-challenge/ {
                root /var/www;

        location / {
                return 301 https://$host$request_uri;

Apply the changes with

service nginx reload

Now we can set up Let’s Encrypt and get a certificate. If you are running Ubuntu 16.04 or later, you can use the official package:

apt-get update
apt-get install letsencrypt
letsencrypt certonly --webroot -w /var/www -d

For other operating systems you’ll need to get certbot installed manually and issue the cert that way:

mkdir /var/letsencrypt
cd /var/letsencrypt
chmod a+x certbot-auto
certbot-auto certonly --webroot -w /var/www -d

Any errors? Did you get your certificate? If so, let’s proceed!

:alarm_clock: You must set a reminder run to letsencrypt renew before your cert expires in three months. For tips on automating this, see the certbot documentation.

Let’s edit the Nginx config again to add HTTPS support:

nano /etc/nginx/sites-available/default 

The old server block in the file needs to stay – it redirects all your users to HTTPS. We need to add a new server block:

server {
  listen 443 ssl http2;  listen [::]:443 ssl http2;
  server_name;  # <-- change this

  ssl on;
  ssl_certificate      /etc/letsencrypt/live/;
  ssl_certificate_key  /etc/letsencrypt/live/;

  ssl_protocols TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  add_header Strict-Transport-Security "max-age=63072000;";
  ssl_stapling on;
  ssl_stapling_verify on;

  client_max_body_size 0;

  location / {
    proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;

If you’re using a version of nginx that cannot speak HTTP2 yet, remove http2 above (twice). The version of nginx offered in Ubuntu 16.04.1 can handle HTTP2.

:warning: Remember to modify the site name and paths as needed above. If you modified the volumes: section of your app.yml, you’ll also need to change the proxy_pass line.

Let’s reload the config we just changed:

service nginx reload

Your site should now be back up, securely over HTTPS. :tada:

Now tell Discourse that it should always use HTTPS URLs by checking use https in Site Settings.

:bell: We recommend that you run SSL Server Test (Powered by Qualys SSL Labs) on your site to verify that your https settings are secure and safe.

Create an error page

Next, you’ll have to design an error page to show when Discourse is offline. Let’s create a path for it.

mkdir /var/www/errorpages
  • If you’re a talented designer, feel free to build a beautiful page yourself, and share it here!
  • If you need to use external resources like images, load them from /errorpages/.
  • I recommend that you include <meta http-equiv="refresh" content="120"> in your page – this will refresh the page every 120 seconds, which means that Discourse will load automatically once it’s available again.
  • Name your main HTML file discourse_offline.html, and place all files in /var/www/errorpages/.

If you’re happy with a page made by an untalented designer, you can simply steal my design instead :blush:

discourse_offline.html (1.9 KB)
d-logo-sketch.png (14 KB)
sob.png (1 KB)

Once you’re done, edit the Nginx config to serve the page you just created:

nano /etc/nginx/sites-available/default 

Simply add

location /errorpages/ {
    alias /var/www/errorpages/;

to the HTTPS server section of your nginx config, and then reload.

service nginx reload

Test by visiting in your browser – you should see your new error page :no_entry:

Serve your error page

Finally, we’ll set up nginx to serve your error page when it cannot reach Discourse, or when Discourse isn’t ready yet. Add the following lines to the location / block:

error_page 502 =502 /errorpages/discourse_offline.html;
proxy_intercept_errors on;

Apply your changes, as usual:

service nginx reload

If you want to test your settings, run

cd /var/discourse
./launcher stop app

to take your site offline and try to visit it:

Don’t forget to run

cd /var/discourse
./launcher start app

afterwards so your Discourse is available again!

A better "site not available" page
Site maintenance mode during rebuilds?
Why my discourse's syster always block users itself?
Site unreachable after adding RapidSSL certificate
Rate Limiting when behind Nginx Proxy
I don't underestand discourse topic/post change history
Subfolder with SSL and nginx reverse proxy
Nginx / letsencrypt / docker -- Ubuntu 16.04 - [FIXED] - [Solution linked in topic]
Idea when Upgrading your discourse
Why are my 301 redirects not working? IPv6 Users also show as localhost
How should I enable letsencrypt while discourse is beside other websites
How should I enable letsencrypt while discourse is beside other websites
Discourse High Availability
Nginx / letsencrypt / docker -- Ubuntu 16.04 - [FIXED] - [Solution linked in topic]
How to set set_real_ip_from for discourse
Let's Encrypt won't renew with offline page
Issue Onebox in HTTPS
How to make discourse only reachable from two IP adresses?
LinkedIn OAuth2 Plugin
Twitter, Github and normal username and password logins not working after upgrade
Next selection shortcut key J jumps from first post to bottom of page
Discourse embedding still has some bugs?
Favicon failing to load - false positive with 1.8.0.beta9+20?
"Down for Maintenance" page
(Jeff Atwood) #2

To make the cert auto renew in Ubuntu 14.04 LTS, I did this:

cd /etc/cron.daily
nano letsencrypt-renew

and added to the file this renew command, taken from the certbot docs:

/var/letsencrypt/certbot-auto renew --quiet --no-self-upgrade

I then had to change permissions on the file so it could run:

chmod 755 letsencrypt-renew

After that I was able to run the bash script manually and it seemed to work! It’s only daily, the docs recommend twice a day, but once a day seems like enough renewal opportunities to me, since the certs expire every 90 days.

(Sam Saffron) #3

One big issue here is that you will be killing http2 support on latest chrome unless you a rerunning absolute latest Ubuntu, which you are not.

(Gerhard Schlager) #4

Why not put the reverse proxy (nginx with the latest openssl and let’s encrypt) in another Docker container? :wink:

(Sam Saffron) #5

Personally if I wanted the best experience

  • I would add an haproxy in tcp mode container to farm reqs to either online/offline container

  • have the nginx offline container mount the same SSL volume so you don’t need to fuss with running lets encrypt in cron and simply have existing template run it

  • have a data container so I can bootstrap while online

nginx proxying to nginx is tricky to get right, for example the setup here is restricting uploads to 2 megs, but plenty of other caveats

(Felix Freiberger) #6

Why is that? I have a super similar setup running in production, and just successfully uploaded a 10 MB file to it.

(Sam Saffron) #7

Correction, 20mb :slight_smile: but really you want to set that to 0

(Felix Freiberger) #8

Good suggestion, I’ve done that.
(I also made the post a wiki post.)

(Sam Saffron) #9

Honestly, I would be significantly less worried about this stuff if the offline/proxy app was running in a container, so much less error prone

I worry that there is way too many manual steps here and underlying versions are unknown

(Felix Freiberger) #10

You may be right – but to my defense: We’ve been recommending a setup like this for running other sites on the same host for two years now.

(Sam Saffron) #11

In my defence that howto is using “listen spdy” which kind of proves my point :slight_smile:

Cc @riking

(Jeff Atwood) #12

So it looks like @sam is pretty strongly opposed to the current approach, and looking at how difficult it is to get a newly compiled Nginx on Ubuntu 14.04 with OpenSSL 1.0.2 support (absolutely required for http/2 to work), I tend to agree. @mpalmer says Ubuntu 14 will never get a new version of OpenSSL in any form via apt-get, and 1.0.1 is end of life in December of this year.

(Is it possible to copy the nginx binary from Ubuntu 16 to Ubuntu 14? Does that work? That might be simplest, if it does…)

Anyway, it looks like we need another container here, a container that holds Nginx to do the routing and so on.

(Matt Palmer) #13

Doesn’t look like it; the package dependency for, ironically, libssl1.0.0 is the sticking point. It might work, if the ABI extensions in libssl1.0.0 aren’t relied upon, but it’d be a risky thing to try in general. I recommend a separate nginx-only container.

(Jeff Atwood) #14

This change seems to cause problems with https redirects for logins, too – even with use https (force https) checked. After instituting the nginx outer layer with https, I had to:

  • enable http return URL for Google logins
  • enable http return URL for GitHub logins

Otherwise you get errors, e.g.

(github) Authentication failure! redirect_uri_mismatch: OmniAuth::Strategies::OAuth2::CallbackError, redirect_uri_mismatch | The redirect_uri MUST match the registered callback URL for this application. |

These are problems I didn’t have when the site was using Let’s Encrypt inside the container…

(Felix Freiberger) #15

I tried to research that and think this is a Discourse bug.

Enabling use https should cause Discourse to only use HTTPS URLs, but neither the Github authenticator, the Google authenticator nor their superclass inspect this site setting. Omniauth tries to detect SSL like this:


proxy_set_header X-Forwarded-Proto $scheme;

to the nginx configuration might work around that, but I’m not sure this is passed on by Discourse’s internal nginx and cannot test that right now.

I do know that SSO is not affected.

Scrolling upwards jumps to top post
Not refreshing after social login due to http vs https
(Jeff Atwood) #16

@fefrei I am not sure this is needed, because I just rebuilt using the standard web updater. I had two browser windows open:

  1. Home page of site in one window, in anon mode
  2. /admin/upgrade rebuild process going in one window

I mashed f5 like an insane :monkey_face: in window #1 and at no point during the rebuild was the site unavailable to me as an anonymous user hitting the homepage. Screenshot proof:

So… if there is never an outage during a typical web update… is this complex hack really needed?

(Felix Freiberger) #17

This is in place to serve a message during rebuilding and the following boot-up – web upgrades should always be fine :slight_smile:

(It may not be worth it for a stable stock site. I think it’s definitely worth it if you either already have to use a nginx reverse proxy for another reason, or if you rebuild often, e.g. to install or uninstall plugins.)

(Jeff Atwood) #18

There is some stuff I don’t like about this

  • it is quite complex
  • it screws up http/2 completely (unless you are on Ubuntu 16)
  • it does not help in the typical update case only in the rare full rebuild case

(Felix Freiberger) #19

I agree, as long as this is the only reason for a front-end nginx instance.

If you need to have that for some other reason, you can start with the Create an error page section – the rest is really straightforward. (And if you need to set up a front-end nginx, I think these steps are easier to follow than these instructions.)

(Harkirat Singh) #20

Should I delete the rest of the template already included in /etc/nginx/sites-available/default