My ./launcher rebuild app step is failing with the following error
[root@osestaging1 discourse]# time ./launcher rebuild app
...
+ /bin/docker start app
Error response from daemon: container is marked for removal and cannot be started
Error: failed to start containers: app
...
real 10m40.584s
user 0m2.099s
sys 0m1.925s
[root@osestaging1 discourse]#
I’ve done some googling on this issue, and I couldn’t find a definite answer. The most common suggestion was to reboot the server. Yeah, if that’s the “solution” then Discourse is not production-ready.
How can I recover from this without restarting the server?
In my case, I found that–for whatever reason–there was one docker container with a “Dead” Status. Removing that “Dead” container followed by a ./launcher start app worked!
Note that I also found that restarting the docker service and even restarting the server did not resolve this issue. Only the following did:
First let’s get a list of all the docker containers. This is a staging box I’m using for a Discourse POC for our org, and it appears to have accumulated a ton of Discourse-related containers. One stands out with a Status = “Dead”
[root@osestaging1 discourse]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5cc0db30940b app "/bin/bash -c 'cd /p…" 13 hours ago Exited (1) 13 hours ago thirsty_borg
f8733eb8d9e4 684c8db14460 "/sbin/boot" 18 hours ago Dead app
24a1f9f4c038 6a959e2d597c "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago peaceful_leavitt
6932865cc6a1 6a959e2d597c "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago friendly_grothendieck
fce75ef5ce06 app "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago gifted_booth
03ea184c205e app "/bin/bash" 2 weeks ago Exited (127) 2 weeks ago clever_solomon
6bd5bb0ab7b5 app "whoami" 2 weeks ago Exited (0) 2 weeks ago upbeat_booth
4fbcfcc1e05f app "echo hello" 2 weeks ago Created sweet_lalande
88d916eb12b0 app "echo hello" 2 weeks ago Created goofy_allen
4a3b6e123460 app "/bin/bash" 3 weeks ago Exited (1) 3 weeks ago adoring_mirzakhani
ef4f90be07e6 app "/bin/bash" 3 weeks ago Exited (0) 3 weeks ago awesome_mcclintock
580c0e430c47 app "/bin/bash" 3 weeks ago Exited (130) 3 weeks ago naughty_greider
4bce62d2e873 app "/usr/bin/apt-get in…" 3 weeks ago Created boring_lehmann
6d4ef0ebb57d app "/usr/bin/apt-get in…" 3 weeks ago Created loving_davinci
4d5c8b2a90e0 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago quizzical_mestorf
34a3f6146a1d app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago epic_williamson
f0a73d8db0db app "iptables -L" 3 weeks ago Created dazzling_beaver
4f34a5f5ee65 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago quizzical_haslett
0980ad174804 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago wonderful_tereshkova
79413047322f app "/usr/bin/apt-get in…" 3 weeks ago Created naughty_proskuriakova
ba00edad459a app "sudo apt-get instal…" 3 weeks ago Created quizzical_burnell
7364dbb52542 app "sudo apt-get instal…" 3 weeks ago Created cocky_bhaskara
9d0e485beba0 app "sudo apt-get instal…" 3 weeks ago Created nervous_greider
75394a9e553f app "/usr/sbin/iptables …" 3 weeks ago Created admiring_cori
8c59607a7b23 app "iptables -L" 3 weeks ago Created silly_buck
92a929061a43 app "bash" 3 weeks ago Exited (0) 3 weeks ago sleepy_cohen
0d4c01df1acb app "bash" 3 weeks ago Exited (0) 3 weeks ago busy_satoshi
3557078bec62 app "/bin/bash -c 'echo …" 3 weeks ago Exited (0) 3 weeks ago busy_sammet
56360e585353 bd5b8ac7ac36 "/bin/sh -c 'apt upd…" 3 weeks ago Exited (100) 3 weeks ago youthful_hermann
53bbee438a5e 9b33df0cef8e "/bin/sh -c 'apt upd…" 4 weeks ago Exited (127) 4 weeks ago awesome_newton
[root@osestaging1 discourse]#
Now we delete the CONTAINER ID found on the “Dead” row:
[root@osestaging1 discourse]# docker rm f8733eb8d9e4
f8733eb8d9e4
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5cc0db30940b app "/bin/bash -c 'cd /p…" 13 hours ago Exited (1) 13 hours ago thirsty_borg
24a1f9f4c038 6a959e2d597c "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago peaceful_leavitt
6932865cc6a1 6a959e2d597c "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago friendly_grothendieck
fce75ef5ce06 app "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago gifted_booth
03ea184c205e app "/bin/bash" 2 weeks ago Exited (127) 2 weeks ago clever_solomon
6bd5bb0ab7b5 app "whoami" 2 weeks ago Exited (0) 2 weeks ago upbeat_booth
4fbcfcc1e05f app "echo hello" 2 weeks ago Created sweet_lalande
88d916eb12b0 app "echo hello" 2 weeks ago Created goofy_allen
4a3b6e123460 app "/bin/bash" 3 weeks ago Exited (1) 3 weeks ago adoring_mirzakhani
ef4f90be07e6 app "/bin/bash" 3 weeks ago Exited (0) 3 weeks ago awesome_mcclintock
580c0e430c47 app "/bin/bash" 3 weeks ago Exited (130) 3 weeks ago naughty_greider
4bce62d2e873 app "/usr/bin/apt-get in…" 3 weeks ago Created boring_lehmann
6d4ef0ebb57d app "/usr/bin/apt-get in…" 3 weeks ago Created loving_davinci
4d5c8b2a90e0 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago quizzical_mestorf
34a3f6146a1d app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago epic_williamson
f0a73d8db0db app "iptables -L" 3 weeks ago Created dazzling_beaver
4f34a5f5ee65 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago quizzical_haslett
0980ad174804 app "/usr/bin/apt-get in…" 3 weeks ago Exited (0) 3 weeks ago wonderful_tereshkova
79413047322f app "/usr/bin/apt-get in…" 3 weeks ago Created naughty_proskuriakova
ba00edad459a app "sudo apt-get instal…" 3 weeks ago Created quizzical_burnell
7364dbb52542 app "sudo apt-get instal…" 3 weeks ago Created cocky_bhaskara
9d0e485beba0 app "sudo apt-get instal…" 3 weeks ago Created nervous_greider
75394a9e553f app "/usr/sbin/iptables …" 3 weeks ago Created admiring_cori
8c59607a7b23 app "iptables -L" 3 weeks ago Created silly_buck
92a929061a43 app "bash" 3 weeks ago Exited (0) 3 weeks ago sleepy_cohen
0d4c01df1acb app "bash" 3 weeks ago Exited (0) 3 weeks ago busy_satoshi
3557078bec62 app "/bin/bash -c 'echo …" 3 weeks ago Exited (0) 3 weeks ago busy_sammet
56360e585353 bd5b8ac7ac36 "/bin/sh -c 'apt upd…" 3 weeks ago Exited (100) 3 weeks ago youthful_hermann
53bbee438a5e 9b33df0cef8e "/bin/sh -c 'apt upd…" 4 weeks ago Exited (127) 4 weeks ago awesome_newton
[root@osestaging1 discourse]#
And now the ./launcher start app works!
[root@osestaging1 discourse]# ./launcher start app
...
+ /bin/docker run ... /sbin/boot
b55581b930865eb4cf744410cdb7dc2f5ce37517042781a6227fbb640b456d86
...
[root@osestaging1 discourse]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b55581b93086 local_discourse/app "/sbin/boot" 7 seconds ago Up 5 seconds app
[root@osestaging1 discourse]#