Bootstrap failed after rebuild - changed base image


(Bruce Becker) #1

Hi all,

Just thought I’d share my experience with you. I had a similar experience when changing the domain name (we have a custom-managed DNS). However, the solution provided by this post is not valid.

The error I got was:

INFO -- : > cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr}.log /var/www/discourse/log"
ln: failed to create symbolic link ‘/var/www/discourse/log/production.log’: File exists
ln: failed to create symbolic link ‘/var/www/discourse/log/production_errors.log’: File exists
ln: failed to create symbolic link ‘/var/www/discourse/log/unicorn.stdout.log’: File exists
ln: failed to create symbolic link ‘/var/www/discourse/log/unicorn.stderr.log’: File exists
I, [2015-05-18T09:59:43.413737 #38]  INFO -- : 
I, [2015-05-18T09:59:43.414153 #38]  INFO -- : Terminating async processes
I, [2015-05-18T09:59:43.414276 #38]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.3/bin/postmaster -D /etc/postgresql/9.3/main pid: 108
I, [2015-05-18T09:59:43.414403 #38]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 231
2015-05-18 09:59:43 UTC [108-2] LOG:  received fast shutdown request
2015-05-18 09:59:43 UTC [108-3] LOG:  aborting any active transactions
2015-05-18 09:59:43 UTC [115-2] LOG:  autovacuum launcher shutting down
231:signal-handler (1431943183) Received SIGTERM scheduling shutdown...
2015-05-18 09:59:43 UTC [112-1] LOG:  shutting down
231:M 18 May 09:59:43.497 # User requested shutdown...
231:M 18 May 09:59:43.497 * Saving the final RDB snapshot before exiting.
231:M 18 May 09:59:43.523 * DB saved on disk
231:M 18 May 09:59:43.523 # Redis is now ready to exit, bye bye...
2015-05-18 09:59:43 UTC [112-2] LOG:  database system is shut down

RuntimeError: cd /var/www/discourse && bash -c "ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr}.log /var/www/discourse/log" failed with return #<Process::Status: pid 299 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"code", "cmd"=>["git reset --hard", "git clean -f", "git remote set-branches --add origin master", "git pull", "git fetch origin $version", "git checkout $version", "mkdir -p tmp/pids", "mkdir -p tmp/sockets", "touch tmp/.gitkeep", "mkdir -p                    /shared/log/rails", "bash -c \"touch -a           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr}.log\"", "bash -c \"ln    -s           /shared/log/rails/{production,production_errors,unicorn.stdout,unicorn.stderr}.log $home/log\"", "bash -c \"mkdir -p           /shared/{uploads,backups}\"", "bash -c \"ln    -s           /shared/{uploads,backups} $home/public\"", "chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups"]}

There is clearly something wrong with the way that existing symlinks are dealt with.
I had tried to be fancy and use an existing docker image as the base (editing launcher) to use it, which didn’t work. However, after cleaning up (rm -rf) everything under /var/discourse/shared I re-ran bootstrap (after removing all images on the host), the bootstrap worked.

I guess the question is : how do I run discourse in a pre-prepared container (ie, move from my laptop dev environment to the hosted web environment ?)

(Kane York) #3

Wait, what? You tried to change the domain name, you found a topic where someone screwed up the indentation in their YML file, and you ended up wiping your database?

Why would you want to do that? The app.yml is supposed to contain all instructions necessary to recreate the container from the base image.

If you want to move your forum from one computer to another, use backups.

(Erik Swanson) #4

For starters, it’d be nice to eliminate the ridiculous downtime while the rebuild is running. I’d also like to cook the image, try it in a staging environment, then push/pull (the same exact image!) down and launch it in a live environment with the only difference being what volumes/environment variables are provided.

(I’m holding out hope that the Discourse team isn’t actually oblivious to the magnitude of anti-patterns present in how Discourse currently uses Docker. :pensive: To be fair, it does work and get the job done… at least until you try to treat it like how you can treat most other Docker images. )

(Sam Saffron) #5

Jeez, have you looked at data and web only templates. discourse_docker/samples at master · discourse/discourse_docker · GitHub use those and you have zero downtime during bootstrap.

Internally we use a private repo for our docker images.

We deploy on discourse docker to 10 different servers with zero downtime so yeah… there is that.

(Erik Swanson) #6

Do you use bootstrap/tag/push/pull/run handling of images for anything other than web-only?

In postgres.template.yml for instance, it looks like there are a lot of side-effects to the /shared volume that happen during the “build.”

(Bruce Becker) #7

My idea of “backups” was to snapshot and move a container. What would you have suggested as a backup ? Thanks !

(Sam Saffron) #8

containers are disposable, they are pretty much the only thing you don’t need to move.

(Bruce Becker) #9

Hey @sam,

So, please excuse my ignorance here, I am just grappling with the different ways of doing things with containers. What I’m used to doing is developing stuff I want, putting that all into the container I want, then moving the container to the place I want it, with some contextualisation.

It seems that discourse’s use of containers does things a bit differently - for one thing, I tried to deploy on my laptop as I mentioned before, expecting things to be done entirely within a series of container. However, I’m finding that there are things which have (from my point of view) “leaked out” of the containers onto my machine - for example the stuff in /var/discourse/.

If I want to move my installation now, I have move the content and the database, whilst I was expecting just to have to snapshot the container(s) I wanted and move those.

“Best practice” rant aside (which I’m not in a position to comment on anyway) - can you point me to the way discourse “expects” to be migrated ?

Thanks !

(Sam Saffron) #10

Have a look at your container config

You explicitly telling Docker that you want all shared data to be stored on the host.

If you wanted to to “transmit” this config to another host you need to scp the important shared data to the same location.

You could configure this with data containers, but keeping this stuff inside the main container is never considered a good practice.

(Bruce Becker) #11

Thanks @sam, that’s just what I was looking for. Much obliged.

(Kane York) #12

I’m taking about the /admin/backups page. You go there, ask Discourse to create a backup of the whole forum, and it gives you a .tar.gz file.

(Bruce Becker) #13

yep, found that after poking about, thanks :smile: Wasn’t aware that it existed… it’s nice that Discourse gives you this easy way of doing backups, however, it doesn’t seem obvious to me how it could be automated… #stilllearning

(Sam Saffron) #14

Enable daily backups you ca even automatically push to s3 or rsync the folder