Issues with rebuilding - AWS Docker

(Gareth Williams) #1

Hey All,

I have managed to import our entire history of VBulletin across to an install of Discourse.

We have it setup via Docker on AWS and all has been going well however, we suddenly have an issue with the site becoming unstable - I have been unable to browse the site intermittently. Now the site isnt running at all and I have been going for the nuclear option, I have tried running a rebuild however, it just sits there.

From what I can see, every time the command is run it spins up a new docker container without taking down the previous one:

bc0aa2a365ed discourse/discourse:1.3.10 "echo working" 27 minutes ago Up 27 minutes keen_mclean
1f668baf3c33 discourse/discourse:1.3.10 "echo working" 40 minutes ago Up 40 minutes gifted_lewin
8818eedd48ae discourse/discourse:1.3.10 "echo working" About an hour ago Up About an hour romantic_gates
42993e97fa78 discourse/discourse:1.3.10 "echo working" 2 hours ago Up 2 hours elegant_goldwasser
1e94ce3fa47f discourse/discourse:1.3.10 "echo working" 3 hours ago Up 3 hours epic_haibt
0e4b77f62034 discourse/discourse:1.3.10 "echo working" 15 hours ago Up 15 hours affectionate_bhabha
075ec3cd4ff2 discourse/discourse:1.3.10 "echo working" 16 hours ago Up 15 hours adoring_pike
4076a433cfbf samsaffron/docker-gc "/docker-gc" 21 hours ago Up 21 hours angry_newton
a8d0cf73d03a local_discourse/app "/sbin/boot" 27 hours ago Up 21 hours>80/tcp,>443/tcp app

As you will see the last few containers all seem to have halted when running an “echo working” command. Is this something anyone has come across at all? Any Ideas on what I should be trying?

(Matt Palmer) #2

Looks like Docker has got itself hosed. Logs, strace, etc might provide clues.

(Gareth Williams) #3

Hey @mpalmer

Thanks mate - will have a dig and see what i can find!

(Rafael dos Santos Silva) #4

Just keep in mind that the image that actually run Discourse for you is the last one, local_discourse/app, so search for logs there.

Also a ./launcher rebuild app should give some info.

(Gareth Williams) #5

Hey All,

Really weird - I decided to go nuclear - restarted the server and all seems to be back.

Im going to do a rebuild now which appears to be moving - think i might be ok here.

(Andrew Waugh) #6

The docker images with “echo working” are a result of running ./launcher rebuild app.

./launcher rebuild app does (roughly speaking) the following:

  1. Some housekeeping (git pull /var/discourse, check docker version, storage)
  2. Stop the existing container
  3. download the latest container
  4. bootstrap the downloaded container
  5. set the docker run commands for the downloaded and bootstrapped container
  6. change the id of the new container to local_discourse/app
  7. run the container local_discourse/app

I suspect that all of your “discourse/discourse:1.3.10 “echo working”” images are zombies or rather corpses left over from a “rebuild app” which has died (ran out of diskspace, swapped itself to death, interrupted via ctrl-z, problems in app.yml)

On a plain standalone install you normally only have 3 images:

  1. local_discourse/app (The image you are currently running)
  2. discourse/discourse (The image your current instance was bootstrapped from)
  3. samsaffron/docker-gc (Sam’s Garbage collector, which apparently participates in the magic above)

To clean things up (presuming you are only running discourse) :

cd /var/discourse
docker ps (you should only see one container running)
./launcher cleanup (will ask you if it should remove all the unused containers)

(Gareth Williams) #7

thanks @JagWaugh really helpful for me in future!

All is cleaned up and running nicely now :slight_smile: