Docker upgrade prevents access to Discourse on Digital Ocean


(Susan Spencer) #1

I upgraded my Discourse software today. It is installed on a DigitalOcean droplet with Ubuntu 14.04
The upgrade included an upgrade to Docker.

After upgrade, no-one can access the forum, they get 502 Bad Gateway nginx error.
Restarting the droplet did not fix the error.

Looking into this problem, found that nginx service had not started automatically, and after trying several methods I could not get nginx started manually.
In the /var/log/nginx/error.log this message occurred several times:
[emerg] 21412#0: bind() to 0.0.0.0:80 failed (98:Address already in use)
So looking into this further, this appears to be related to a Docker issue, opened two days ago, related to upgrading to the latest version of Docker. This Docker issue describes upgrading on AWS with Ubuntu, and after the upgrade nginx cannot start because port 80 is detected to be in use, but it actually isn’t.
https://github.com/docker/docker/issues/25877


(Kane York) #2

You need to follow the guide here:


(Sam Saffron) #3

We run 1.12.0 throughout our entire prd environment. I just upgraded samsaffron.com to 1.12.1, have not seen this issue previously.

Can you include the output of docker info on your server?

Is there any web server listening on port 80 on the host somehow?

Are you using the letsencrypt SSL template?


(Jay Pfaffman) #4

Maybe ./launcher rebuild app again?


(Susan Spencer) #5

According to this Docker thread on github:
Error when try to start nginx container (Bind for 0.0.0.0:80 failed: port
is already allocated.) #25877
Error when try to start nginx container (Bind for 0.0.0.0:80 failed: port is already allocated.) · Issue #25877 · moby/moby · GitHub,
one user fixed this by rolling back to a previous version of Docker. He:
rolled back to Docker 1.11.2, reinstalled 1.12.0 (not 1.12.1), added
0.0.0.0 to the port binding, retarted his system, and everything is working.
I don’t know how to roll back to a previous version of Docker on Digital
Ocean.

Kane, The only website running on my droplet is the Discourse forum,
forum.valentina-project.org

Sam, here is the output of docker info:

I am following instructions here to see if everything is working correctly:
https://www.digitalocean.com/community/tutorials/how-to-
install-discourse-behind-nginx-on-ubuntu-14-04,

Here is the result from
./letsencrypt-auto certonly --standalone --email susan@valentina-project.org
–agree-tos -d forum.valentina-project.org


(Kane York) #6

Oh, I missed that and assumed the problem was that a second nginx (not inside the container) was failing to start. Sorry.


(Sam Saffron) #7

Very interesting, your docker info is surprisingly close to the one that is running on my instance so there is nothing really fishy going on with your install.

To be clear though, is this an HTTP only setup or are you using our letsencrypt template.

Regarding rolling back … you can always uninstall docker-engine and install a specific version:

apt-get install docker-engine=1.12.0-0~trusty

Or something along those lines.


(Susan Spencer) #8

I restored our droplet from backup to remove the upgrade. Seems fine now. We’re looking into how to prevent the occasional 504 nginx bad gateway error now, which is a separate issue involving the same set of software.