How to fix "container is marked for removal and cannot be started" without reboot

In my case, I found that–for whatever reason–there was one docker container with a “Dead” Status. Removing that “Dead” container followed by a ./launcher start app worked!

Note that I also found that restarting the docker service and even restarting the server did not resolve this issue. Only the following did:

First let’s get a list of all the docker containers. This is a staging box I’m using for a Discourse POC for our org, and it appears to have accumulated a ton of Discourse-related containers. One stands out with a Status = “Dead”

[root@osestaging1 discourse]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
5cc0db30940b        app       "/bin/bash -c 'cd /p…"   13 hours ago        Exited (1) 13 hours ago                        thirsty_borg
f8733eb8d9e4        684c8db14460        "/sbin/boot"             18 hours ago        Dead                                           app
24a1f9f4c038        6a959e2d597c        "/bin/bash"              2 weeks ago         Exited (1) 2 weeks ago                         peaceful_leavitt
6932865cc6a1        6a959e2d597c        "/bin/bash"              2 weeks ago         Exited (1) 2 weeks ago                         friendly_grothendieck
fce75ef5ce06        app       "/bin/bash"              2 weeks ago         Exited (0) 2 weeks ago                         gifted_booth
03ea184c205e        app       "/bin/bash"              2 weeks ago         Exited (127) 2 weeks ago                       clever_solomon
6bd5bb0ab7b5        app       "whoami"                 2 weeks ago         Exited (0) 2 weeks ago                         upbeat_booth
4fbcfcc1e05f        app       "echo hello"             2 weeks ago         Created                                        sweet_lalande
88d916eb12b0        app       "echo hello"             2 weeks ago         Created                                        goofy_allen
4a3b6e123460        app       "/bin/bash"              3 weeks ago         Exited (1) 3 weeks ago                         adoring_mirzakhani
ef4f90be07e6        app       "/bin/bash"              3 weeks ago         Exited (0) 3 weeks ago                         awesome_mcclintock
580c0e430c47        app       "/bin/bash"              3 weeks ago         Exited (130) 3 weeks ago                       naughty_greider
4bce62d2e873        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        boring_lehmann
6d4ef0ebb57d        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        loving_davinci
4d5c8b2a90e0        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         quizzical_mestorf
34a3f6146a1d        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         epic_williamson
f0a73d8db0db        app       "iptables -L"            3 weeks ago         Created                                        dazzling_beaver
4f34a5f5ee65        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         quizzical_haslett
0980ad174804        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         wonderful_tereshkova
79413047322f        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        naughty_proskuriakova
ba00edad459a        app       "sudo apt-get instal…"   3 weeks ago         Created                                        quizzical_burnell
7364dbb52542        app       "sudo apt-get instal…"   3 weeks ago         Created                                        cocky_bhaskara
9d0e485beba0        app       "sudo apt-get instal…"   3 weeks ago         Created                                        nervous_greider
75394a9e553f        app       "/usr/sbin/iptables …"   3 weeks ago         Created                                        admiring_cori
8c59607a7b23        app       "iptables -L"            3 weeks ago         Created                                        silly_buck
92a929061a43        app       "bash"                   3 weeks ago         Exited (0) 3 weeks ago                         sleepy_cohen
0d4c01df1acb        app       "bash"                   3 weeks ago         Exited (0) 3 weeks ago                         busy_satoshi
3557078bec62        app       "/bin/bash -c 'echo …"   3 weeks ago         Exited (0) 3 weeks ago                         busy_sammet
56360e585353        bd5b8ac7ac36        "/bin/sh -c 'apt upd…"   3 weeks ago         Exited (100) 3 weeks ago                       youthful_hermann
53bbee438a5e        9b33df0cef8e        "/bin/sh -c 'apt upd…"   4 weeks ago         Exited (127) 4 weeks ago                       awesome_newton
[root@osestaging1 discourse]# 

Now we delete the CONTAINER ID found on the “Dead” row:

[root@osestaging1 discourse]# docker rm f8733eb8d9e4
f8733eb8d9e4
[root@osestaging1 discourse]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
5cc0db30940b        app       "/bin/bash -c 'cd /p…"   13 hours ago        Exited (1) 13 hours ago                        thirsty_borg
24a1f9f4c038        6a959e2d597c        "/bin/bash"              2 weeks ago         Exited (1) 2 weeks ago                         peaceful_leavitt
6932865cc6a1        6a959e2d597c        "/bin/bash"              2 weeks ago         Exited (1) 2 weeks ago                         friendly_grothendieck
fce75ef5ce06        app       "/bin/bash"              2 weeks ago         Exited (0) 2 weeks ago                         gifted_booth
03ea184c205e        app       "/bin/bash"              2 weeks ago         Exited (127) 2 weeks ago                       clever_solomon
6bd5bb0ab7b5        app       "whoami"                 2 weeks ago         Exited (0) 2 weeks ago                         upbeat_booth
4fbcfcc1e05f        app       "echo hello"             2 weeks ago         Created                                        sweet_lalande
88d916eb12b0        app       "echo hello"             2 weeks ago         Created                                        goofy_allen
4a3b6e123460        app       "/bin/bash"              3 weeks ago         Exited (1) 3 weeks ago                         adoring_mirzakhani
ef4f90be07e6        app       "/bin/bash"              3 weeks ago         Exited (0) 3 weeks ago                         awesome_mcclintock
580c0e430c47        app       "/bin/bash"              3 weeks ago         Exited (130) 3 weeks ago                       naughty_greider
4bce62d2e873        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        boring_lehmann
6d4ef0ebb57d        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        loving_davinci
4d5c8b2a90e0        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         quizzical_mestorf
34a3f6146a1d        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         epic_williamson
f0a73d8db0db        app       "iptables -L"            3 weeks ago         Created                                        dazzling_beaver
4f34a5f5ee65        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         quizzical_haslett
0980ad174804        app       "/usr/bin/apt-get in…"   3 weeks ago         Exited (0) 3 weeks ago                         wonderful_tereshkova
79413047322f        app       "/usr/bin/apt-get in…"   3 weeks ago         Created                                        naughty_proskuriakova
ba00edad459a        app       "sudo apt-get instal…"   3 weeks ago         Created                                        quizzical_burnell
7364dbb52542        app       "sudo apt-get instal…"   3 weeks ago         Created                                        cocky_bhaskara
9d0e485beba0        app       "sudo apt-get instal…"   3 weeks ago         Created                                        nervous_greider
75394a9e553f        app       "/usr/sbin/iptables …"   3 weeks ago         Created                                        admiring_cori
8c59607a7b23        app       "iptables -L"            3 weeks ago         Created                                        silly_buck
92a929061a43        app       "bash"                   3 weeks ago         Exited (0) 3 weeks ago                         sleepy_cohen
0d4c01df1acb        app       "bash"                   3 weeks ago         Exited (0) 3 weeks ago                         busy_satoshi
3557078bec62        app       "/bin/bash -c 'echo …"   3 weeks ago         Exited (0) 3 weeks ago                         busy_sammet
56360e585353        bd5b8ac7ac36        "/bin/sh -c 'apt upd…"   3 weeks ago         Exited (100) 3 weeks ago                       youthful_hermann
53bbee438a5e        9b33df0cef8e        "/bin/sh -c 'apt upd…"   4 weeks ago         Exited (127) 4 weeks ago                       awesome_newton
[root@osestaging1 discourse]# 

And now the ./launcher start app works!

[root@osestaging1 discourse]# ./launcher start app
...
+ /bin/docker run ... /sbin/boot
b55581b930865eb4cf744410cdb7dc2f5ce37517042781a6227fbb640b456d86
...
[root@osestaging1 discourse]# docker ps
CONTAINER ID        IMAGE                           COMMAND             CREATED             STATUS              PORTS               NAMES
b55581b93086        local_discourse/app   "/sbin/boot"        7 seconds ago       Up 5 seconds                            app
[root@osestaging1 discourse]#