Upgrade (running in Docker) Fails


(Deviant) #1

Hi everyone,

The last 3 times that I have had to use:

git pull
./launcher rebuild app

It has resulted in:

+ /usr/bin/docker rm app
Error response from daemon: driver "aufs" failed to remove root filesystem for [redacted] : rename /var/lib/docker/aufs/mnt/[redacted] /var/lib/docker/aufs/mnt/[redacted]-removing: device or resource busy
Error response from daemon: Container is marked for removal and cannot be started.
Error: failed to start containers: app

Is this an issue with Docker, Discourse, or my server?

Every time that this has happened, the only way to get things working again has been to purge docker and remove all mounted file systems and containers then reinstall (at least the discourse data seems to remain intact).
Problem is, I have other software running in Docker as well now. Going through that whole process seems a bit much.

Any ideas?

Appreciate the feedback. Not sure what other info I could give that would be helpful …


(Hosein Naseri) #2

when I face this problem, I just reboot the system and in the next rebuild, I don’t get any error.


(Deviant) #3

Ima try that now, but just before I do – after a reboot:

docker rm [container-id]

Produces:

Error response from daemon: driver "aufs" failed to remove root filesystem for 9b81c0b4b9366349d378b564df606744efc9425d48e9463df2ff6fb98d304f69: no such file or directory

This is the same problem I ran into before. I predict that ./launcher rebuild will fail. Again.


(Darren) #4

I have had the ‘Container is marked for removal and cannot be started’ error multiple times, it requires a restart of docker or a system re-boot. Not sure what causes it but that is how to solve the problem until someone with more knowledge finds a proper fix for the problem.


(Deviant) #5

Thanks Darren

But

(sames as the previous 2 times) - even after rebooting after the first time it failed:

Removing old container
+ /usr/bin/docker rm app
Error response from daemon: driver "aufs" failed to remove root filesystem for 9b81c0b4b9366349d378b564df606744efc9425d48e9463df2ff6fb98d304f69: no such file or directory

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: Container is marked for removal and cannot be started.
Error: failed to start containers: app

Clearly - this is a Docker problem.


(Deviant) #6

So, what I did was

mv container/app.yml container/app1.yml
./launcher rebuild app1

And that seems to have worked. My forum is back online and all the data is intact, however - I now have that dead container that I (still) can’t remove…

docker ps -a
10530f78ba5e        local_discourse/app1   "/sbin/boot"             8 minutes ago       Up 8 minutes                                           app1
9b81c0b4b936        ee71f048c533           "/sbin/boot"             8 days ago          Dead                                                   app

(Dimitris Kritsotakis) #7

I know that will sound weird but have you tried restarting your server? I had a similar problem and restart solved it.