Docker 1.5.0 has issues with iptables

thisgeekza · February 18, 2015, 6:13am

I just upgraded docker to 1.5.0. Since then I have been unable to bootstrap the web container.

$ sudo ./launcher bootstrap web
WARNING: No swap limit support
Calculated ENV: -e LANG=en_US.UTF-8 -e HOME=/root -e RAILS_ENV=production -e UNICORN_WORKERS=6 -e UNICORN_SIDEKIQS=1 -e RUBY_GC_MALLOC_LIMIT=40000000 -e RUBY_HEAP_MIN_SLOTS=800000 -e DISCOURSE_DB_SOCKET= -e DISCOURSE_DB_HOST=data -e DISCOURSE_DB_PORT= -e DISCOURSE_DB_PASSWORD=xxx -e DISCOURSE_REDIS_HOST=data -e DISCOURSE_DEVELOPER_EMAILS=xxx@xxx -e DISCOURSE_HOSTNAME=xxx -e DISCOURSE_SMTP_ADDRESS=xxx -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=xxx -e DISCOURSE_SMTP_PASSWORD=xxx
cd /pups && git pull && /pups/bin/pups --stdin
FATA[0000] Error response from daemon: Cannot start container 880ca22223f382d1ab33cc0c0a7e2c5899f3b9e6bc7cb46aba9238b3610a1183:  (exit status 1)
880ca22223f382d1ab33cc0c0a7e2c5899f3b9e6bc7cb46aba9238b3610a1183
FAILED TO BOOTSTRAP

Downgrading to docker 1.4.1 allows web to bootstrap. Oddly enough, data can bootstrap with 1.5.0, but web won’t.

sam · February 18, 2015, 6:46am

did you reboot post installation, I have been using docker 1.5 on about 15 machines now and it works just fine.

thisgeekza · February 18, 2015, 10:53am

Yes, I did.
Not sure what the issue is, but have rolled back for the time being.

I even tried removing all the discourse related images, and let launcher re-pull them.

thisgeekza · February 18, 2015, 11:46am

Damnit, sorry. The container is built from web_only.yml, not the standalone, if that makes any difference.

codinghorror · February 18, 2015, 5:49pm

Yeah no issues with Docker 1.5 at all here.

thisgeekza · February 18, 2015, 6:01pm

Damn. Then I don’t know what is different about my system that I’m having issues suddenly. It’s a bog standard ubuntu server 14.04 LTS install.

sam · February 18, 2015, 7:58pm

what does docker info return ?

thisgeekza · February 19, 2015, 3:32pm

The one thing that is non-standard about my docker install, is that I don’t like it opening up my containers to the world, so I have added DOCKER_OPTS="--icc=true --iptables=false" to /etc/default/docker.
I then manually add iptables rules to allow the containers to communicate out. I don’t think this is causing the issue with the container not starting or even bootstrapping on 1.5.0 though. Everything works just fine on 1.4.0.

I’m a bit stumped with what’s going on here.

If nobody can give see any issues with what I’m doing, I’ll try set the daemon to default startup params and see if it’s still having an issue. Last resort will be to upgrade the OS to 14.10 and see if it makes any difference.

This is what happens after upgrading docker to 1.5.0 and rebooting:

Docker Info:

Containers: 4
Images: 42
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 53
Execution Driver: native-0.2
Kernel Version: 3.13.0-45-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 31.29 GiB
Name: nephilim
ID: 5R4W:3547:JCVF:OAML:6SGH:6FQL:EUKZ:QRCI:NENU:LAEC:FRXD:MOKS
WARNING: No swap limit support

docker ps -a

CONTAINER ID        IMAGE                               COMMAND                CREATED             STATUS                       PORTS                                                                                                                          NAMES
e3c6c3f66a2c        local_discourse/web:latest          "/sbin/boot"           32 hours ago        Exited (143) 6 minutes ago                                                                                                                                  web
f77454d020ee        local_discourse/data:latest         "/sbin/boot"           33 hours ago        Up 4 minutes                 0.0.0.0:2223->22/tcp, 0.0.0.0:5432->5432/tcp, 0.0.0.0:6379->6379/tcp

The data container auto-starts with no issue. Web throws a flappy.

Try rebuild:

user@nephilim:/var/discourse$ sudo git pull
Already up-to-date.
user@nephilim:/var/discourse$ sudo ./launcher rebuild web
WARNING: No swap limit support
Updating discourse docker
Already up-to-date.
Stopping old container
e3c6c3f66a2ce58bffff438259c87277cbc70d0ac717f249b63de5dcf791219e
Calculated ENV: -e LANG=en_US.UTF-8 -e HOME=/root -e RAILS_ENV=production -e UNICORN_WORKERS=6 -e UNICORN_SIDEKIQS=1 -e RUBY_GC_MALLOC_LIMIT=40000000 -e RUBY_HEAP_MIN_SLOTS=800000 -e DISCOURSE_DB_SOCKET= -e DISCOURSE_DB_HOST=data -e DISCOURSE_DB_PORT= -e DISCOURSE_DB_PASSWORD=xxx -e DISCOURSE_REDIS_HOST=data -e DISCOURSE_DEVELOPER_EMAILS=xxx -e DISCOURSE_HOSTNAME=xxx -e DISCOURSE_SMTP_ADDRESS=xxx -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=xxx -e DISCOURSE_SMTP_PASSWORD=xxx
cd /pups && git pull && /pups/bin/pups --stdin
FATA[0000] Error response from daemon: Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8
FAILED TO BOOTSTRAP

Here’s a bit from upstart/docker.log, not sure if this is helpful at all:

INFO[0343] -job attach(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c) = OK (0)
INFO[0343] POST /v1.17/containers/51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c/wait
INFO[0343] +job wait(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c)
INFO[0343] -job wait(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c) = OK (0)
INFO[0343] GET /v1.17/containers/51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c/json
INFO[0343] +job container_inspect(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c)
INFO[0343] -job container_inspect(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c) = OK (0)
INFO[0343] DELETE /v1.17/containers/51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c?v=1
INFO[0343] +job rm(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c)
INFO[0343] +job log(destroy, 51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c, samsaffron/discourse:1.0.7)
INFO[0343] -job log(destroy, 51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] -job rm(51c945b878e65c39e367af42b4f6847a95a27c4882cb676ce8deb7f39929579c) = OK (0)
INFO[0343] POST /v1.17/containers/create
INFO[0343] +job create()
INFO[0343] +job log(create, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7)
INFO[0343] -job log(create, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] -job create() = OK (0)
INFO[0343] POST /v1.17/containers/ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e/attach?stdin=1&stdout=1&stream=1
INFO[0343] +job container_inspect(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] -job container_inspect(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] +job attach(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] POST /v1.17/containers/ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e/start
INFO[0343] +job start(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] +job allocate_interface(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] -job allocate_interface(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] +job log(start, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7)
INFO[0343] -job log(start, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] -job start(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] +job log(die, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7)
INFO[0343] -job log(die, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] +job release_interface(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] -job release_interface(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] -job attach(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] POST /v1.17/containers/ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e/wait
INFO[0343] +job wait(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] -job wait(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] GET /v1.17/containers/ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e/json
INFO[0343] +job container_inspect(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] -job container_inspect(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] DELETE /v1.17/containers/ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e?v=1
INFO[0343] +job rm(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e)
INFO[0343] +job log(destroy, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7)
INFO[0343] -job log(destroy, ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] -job rm(ede95d1be077d962c39e451f787d8c0ea2c4d8b5e4357fea3fd43d3c7315c63e) = OK (0)
INFO[0343] POST /v1.17/containers/create
INFO[0343] +job create()
INFO[0343] +job log(create, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7)
INFO[0343] -job log(create, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7) = OK (0)
INFO[0343] -job create() = OK (0)
INFO[0343] POST /v1.17/containers/ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8/attach?stderr=1&stdin=1&stdout=1&stream=1
INFO[0343] +job container_inspect(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] -job container_inspect(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)
INFO[0343] +job attach(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] POST /v1.17/containers/ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8/start
INFO[0343] +job start(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] +job allocate_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] -job allocate_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)
INFO[0343] +job link(-A)
iptables failed: iptables --wait -t filter -A DOCKER -i docker0 -o docker0 -p tcp -s 172.17.0.12 -d 172.17.0.2 --dport 22 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)
INFO[0343] -job link(-A) = ERR (1)
INFO[0343] +job link(-D)
INFO[0343] -job link(-D) = OK (0)
INFO[0343] +job release_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] -job release_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)
INFO[0343] +job log(die, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7)
INFO[0343] -job log(die, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7) = OK (0)
Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
INFO[0343] -job start(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = ERR (1)
ERRO[0343] Handler for POST /containers/{name:.*}/start returned error: Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
ERRO[0343] HTTP Error: statusCode=500 Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
INFO[0343] DELETE /v1.17/containers/ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8
INFO[0343] +job rm(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0344] +job log(destroy, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7)
INFO[0344] -job log(destroy, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7) = OK (0)
INFO[0344] -job rm(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)

riking · February 19, 2015, 8:47pm

There’s your failure. It’s the launcher following this part of the config:

## which TCP/IP ports should this container expose?
expose:
  - "80:80"   # fwd host port 80   to container port 80 (http)
  - "2222:22" # fwd host port 2222 to container port 22 (ssh)

thisgeekza · February 20, 2015, 5:36am

I have 4 containers in total, 3 of them start up and run with no issues. All of them have exposed ports.

riking · February 20, 2015, 6:22am

Well, it’s definitely the automatic iptables causing the issue.

Or maybe you added some forwarding rules yourself?

Quoting the relevant part of the log, one whole Docker API request:

INFO[0343] POST /v1.17/containers/ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8/start
INFO[0343] +job start(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] +job allocate_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] -job allocate_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)
INFO[0343] +job link(-A)
iptables failed: iptables --wait -t filter -A DOCKER -i docker0 -o docker0 -p tcp -s 172.17.0.12 -d 172.17.0.2 --dport 22 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)
INFO[0343] -job link(-A) = ERR (1)
INFO[0343] +job link(-D)
INFO[0343] -job link(-D) = OK (0)
INFO[0343] +job release_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8)
INFO[0343] -job release_interface(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = OK (0)
INFO[0343] +job log(die, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7)
INFO[0343] -job log(die, ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8, samsaffron/discourse:1.0.7) = OK (0)
Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
INFO[0343] -job start(ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8) = ERR (1)
ERRO[0343] Handler for POST /containers/{name:.*}/start returned error: Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)
ERRO[0343] HTTP Error: statusCode=500 Cannot start container ed1c0d1c8dc38588cef767b4a70452637e0379c47db74399f81909cde7fcf8d8:  (exit status 1)

thisgeekza · February 20, 2015, 10:44am

Thanks, you are correct. It is that causing the issue.

I removed my DOCKER_OPTIONS to revert it back to default, upgraded back to lxc-docker-1.5.0 and everything works.

Happily, it seems that docker has fixed the annoyance of messing with iptables and exposing all the containers to the wide world, so I no longer need to mess with the DOCKER_OPTIONS and my own iptables rules!

Thanks for everybody’s time and input in this thread, I really appreciate it.

Topic		Replies	Views
Discourse failing to start due to docker/iptables issue Installation docker	1	53	July 5, 2024
Failed to install Discourse in Ubuntu14.04: docker problems Installation docker	19	8355	January 25, 2015
Failed upgrade from 2.5.0.beta3 to 2.5.0.beta4 Installation	4	502	July 16, 2024
Device "docker0" does not exist Installation	36	20381	April 10, 2019
Can't get discourse working again Installation	8	1669	February 16, 2020

Docker 1.5.0 has issues with iptables

Related topics