Better cleanup script


(Zsolt Ero) #1

OK, so I’m running Discourse on a 10 GB OVH VPS and I had to look into how to clean-up Ubuntu properly.

Here is the script I’ve put in an overnight cronjob:

#!/usr/bin/env bash

space=$(df -h /var/lib | awk '{ print $4 }' | tail -1)
echo "Starting Cleanup (bytes free $space)"

# needs apt-get install byobu
purge-old-kernels -y

apt-get autoremove --purge -y

# remove packages with leftover configuration files
dpkg --list |grep "^rc" | cut -d " " -f 3 | xargs -r dpkg --purge

apt-get autoclean -y

# docker cleanup
if [ -f /usr/bin/docker ]; then
    docker ps -q -f status=exited | xargs -r docker rm
    docker images -q -f dangling=true | xargs -r docker rmi --force
fi

space=$(df -h /var/lib | awk '{ print $4 }' | tail -1)
echo "Finished Cleanup (bytes free $space)"

I didn’t see any reason why docker-gs should be used, instead of a command line xargs solution. The one they mention on their website “we take extra care to not remove any image tags (e.g., ubuntu:14.04, busybox, etc) that are in use by containers” do not fit our use case, or does it?

Also, I found out that ncdu is an amazing terminal based free space explorer!


(Jeff Atwood) #2

Use ./launcher cleanup instead of the last bits, is my only suggestion.

I also think apt-get autoclean removes old kernels; I know it does on the installs I look at. So I am not sure you need purge-old-kernels there.


(Zsolt Ero) #3

I had some gigabytes of old kernels left there, even after running autoremove and autoclean. Only purge-old-kernels was able to detect and remove them.

What does launcher cleanup do what the two xargs line doesn’t?


(Jay Pfaffman) #4

In a script it’d need to be

echo Y | ./launcher cleanup

(Zsolt Ero) #5

Yes, I opened the file just to check if there is a -y or similar flag supported but I found none.

But still, I’m really interested what is the reason to use an outside dependency, especially a whole docker container when we have the two xargs commands? I really don’t see any reason not to use the xargs commands.


(Roland) #6

when I type the ./launcher cleanup i get this

flag provided but not defined: --format
See ‘/usr/bin/docker info --help’.
You have less than 5GB of free space on the disk. You will need more space to continue

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)

Any ideas?


(cpradio) #7

Type ‘y’? (and more characters)


(Roland) #8

OK I wasnt sure if saying yes would cause any other issues


(Roland) #9

heres the output

docker: ‘system’ is not a docker command.
See ‘docker --help’.
If the cleanup was successful, you may try again now


(cpradio) #10

If you run it again, did it clean anything up? Or do you get the less that 5 GB message again?


(Roland) #11

Yes same thing over again


(cpradio) #12

What version of Docker do you have installed?

docker -v

My sandbox has version 17.05 and running docker system prune gives me a prompt

WARNING! This will remove:
	- all stopped containers
	- all volumes not used by at least one container
	- all networks not used by at least one container
	- all dangling images
Are you sure you want to continue? [y/N]

Which is what the cleanup command should be running for you:


(Roland) #13

1.10.3 build 20f81dd


(cpradio) #14

Wow, you are on a very out-dated version of docker, you should do this:

wget -qO- https://get.docker.com/ | sh

(Roland) #15

ok thanks. I got the digital ocean setup and thats what they did. Any other things I should do when I do that? I have alot of members online right now and site was messing up. Will it take it down again? Or should i wait until when its downtime on the forum?


(cpradio) #16

I’d wait for downtime. I think it would be safe butI don’t want to mislead you in case I’m wrong.

After upgrading docker the cleanup command should work


(Roland) #17

I just tried to rebuild app. failed

FAILED

Pups::ExecError: cd /var/www/discourse && su discourse -c ‘bundle exec rake db:migrate’ failed with return #<Process::Status: pid 536 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn’
exec failed with the params {“cd”=>"$home", “hook”=>“bundle_exec”, “cmd”=>[“su discourse -c ‘bundle install --deployment --verbose --without test --without development’”, “su discourse -c ‘bundle exec rake db:migrate’”, “su discourse -c ‘bundle exec rake assets:precompile’”]}
e7887c0baadfcf4c0eb87b9e2d32fd6acb5d8c33b3d81536c90fce8e65124d64
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one


(cpradio) #18

Can you scroll up? That isn’t specific enough


(Roland) #19
    > 0:  bundler (1.15.3) from /usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.15.3/lib/bundler/source
    > Using byebug 9.0.6
    > 0:  byebug (9.0.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/byebug-9.0.6.gemspec
    > Using certified 1.0.0
    > 0:  certified (1.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/certified-1.0.0.gemspec
    > Using coderay 1.1.1
    > 0:  coderay (1.1.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/coderay-1.1.1.gemspec
    > Using concurrent-ruby 1.0.5
    > 0:  concurrent-ruby (1.0.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/concurrent-ruby-1.0.5.gemspec
    > Using connection_pool 2.2.1
    > 0:  connection_pool (2.2.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/connection_pool-2.2.1.gemspec
    > Fetching cppjieba_rb 0.3.0
    > Installing cppjieba_rb 0.3.0 with native extensions
    > Building native extensions.  This could take a while...
    > 0:  cppjieba_rb (0.3.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/cppjieba_rb-0.3.0.gemspec
    > Using safe_yaml 1.0.4
    > 0:  safe_yaml (1.0.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/safe_yaml-1.0.4.gemspec
    > Using crass 1.0.2
    > 0:  crass (1.0.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/crass-1.0.2.gemspec
    > Using diff-lcs 1.3
    > 0:  diff-lcs (1.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/diff-lcs-1.3.gemspec
    > Using thor 0.19.4
    > 0:  thor (0.19.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/thor-0.19.4.gemspec
    > Fetching discourse_fastimage 2.1.0
    > Installing discourse_fastimage 2.1.0
    > 0:  discourse_fastimage (2.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/discourse_fastimage-2.1.0.gemspec
    > Using exifr 1.2.5
    > 0:  exifr (1.2.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/exifr-1.2.5.gemspec
    > Using fspath 3.1.0
    > 0:  fspath (3.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fspath-3.1.0.gemspec
    > Using image_size 1.5.0
    > 0:  image_size (1.5.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/image_size-1.5.0.gemspec
    > Using in_threads 1.4.0
    > 0:  in_threads (1.4.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/in_threads-1.4.0.gemspec
    > Using progress 3.3.1
    > 0:  progress (3.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/progress-3.3.1.gemspec
    > Fetching email_reply_trimmer 0.1.7
    > Installing email_reply_trimmer 0.1.7
    > 0:  email_reply_trimmer (0.1.7) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/email_reply_trimmer-0.1.7.gemspec
    > Using excon 0.56.0
    > 0:  excon (0.56.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/excon-0.56.0.gemspec
    > Using fabrication 2.9.8
    > 0:  fabrication (2.9.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fabrication-2.9.8.gemspec
    > Using fakeweb 1.3.0
    > 0:  fakeweb (1.3.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fakeweb-1.3.0.gemspec
    > Using multipart-post 2.0.0
    > 0:  multipart-post (2.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/multipart-post-2.0.0.gemspec
    > Using fast_blank 1.0.0
    > 0:  fast_blank (1.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fast_blank-1.0.0.gemspec
    > Using fast_xs 0.8.0
    > 0:  fast_xs (0.8.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fast_xs-0.8.0.gemspec
    > Using ffi 1.9.18
    > 0:  ffi (1.9.18) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ffi-1.9.18.gemspec
    > Using flamegraph 0.9.5
    > 0:  flamegraph (0.9.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/flamegraph-0.9.5.gemspec
    > Using gc_tracer 1.5.1
    > 0:  gc_tracer (1.5.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/gc_tracer-1.5.1.gemspec
    > Using guess_html_encoding 0.0.11
    > 0:  guess_html_encoding (0.0.11) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/guess_html_encoding-0.0.11.gemspec
    > Using hashdiff 0.3.4
    > 0:  hashdiff (0.3.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/hashdiff-0.3.4.gemspec
    > Using hashie 3.5.5
    > 0:  hashie (3.5.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/hashie-3.5.5.gemspec
    > Using highline 1.7.8
    > 0:  highline (1.7.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/highline-1.7.8.gemspec
    > Using hiredis 0.6.1
    > 0:  hiredis (0.6.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/hiredis-0.6.1.gemspec
    > Using htmlentities 4.3.4
    > 0:  htmlentities (4.3.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/htmlentities-4.3.4.gemspec
    > Using http_accept_language 2.0.5
    > 0:  http_accept_language (2.0.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/http_accept_language-2.0.5.gemspec
    > Using jwt 1.5.6
    > 0:  jwt (1.5.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/jwt-1.5.6.gemspec
    > Using kgio 2.11.0
    > 0:  kgio (2.11.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/kgio-2.11.0.gemspec
    > Using libv8 5.9.211.38.1 (x86_64-linux)
    > 0:  libv8 (5.9.211.38.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/libv8-5.9.211.38.1-x86_64-linux.gemspec
    > Using rb-fsevent 0.9.8
    > 0:  rb-fsevent (0.9.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rb-fsevent-0.9.8.gemspec
    > Using ruby_dep 1.5.0
    > 0:  ruby_dep (1.5.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ruby_dep-1.5.0.gemspec
    > Using logster 1.2.7
    > 0:  logster (1.2.7) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/logster-1.2.7.gemspec
    > Using lru_redux 1.1.0
    > 0:  lru_redux (1.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/lru_redux-1.1.0.gemspec
    > Using memory_profiler 0.9.8
    > 0:  memory_profiler (0.9.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/memory_profiler-0.9.8.gemspec
    > Using metaclass 0.0.4
    > 0:  metaclass (0.0.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/metaclass-0.0.4.gemspec
    > Using method_source 0.8.2
    > 0:  method_source (0.8.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/method_source-0.8.2.gemspec
    > Using mini_mime 0.1.3
    > 0:  mini_mime (0.1.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mini_mime-0.1.3.gemspec
    > Using mock_redis 0.17.3
    > 0:  mock_redis (0.17.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mock_redis-0.17.3.gemspec
    > Using moneta 1.0.0
    > 0:  moneta (1.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/moneta-1.0.0.gemspec
    > Using multi_json 1.12.1
    > 0:  multi_json (1.12.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/multi_json-1.12.1.gemspec
    > Using multi_xml 0.6.0
    > 0:  multi_xml (0.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/multi_xml-0.6.0.gemspec
    > Using mustache 1.0.5
    > 0:  mustache (1.0.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mustache-1.0.5.gemspec
    > Using oauth 0.5.1
    > 0:  oauth (0.5.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/oauth-0.5.1.gemspec
    > Using oj 3.1.0
    > 0:  oj (3.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/oj-3.1.0.gemspec
    > Using ruby-openid 2.7.0
    > 0:  ruby-openid (2.7.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ruby-openid-2.7.0.gemspec
    > Using redis 3.3.3
    > 0:  redis (3.3.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/redis-3.3.3.gemspec
    > Using pg 0.20.0
    > 0:  pg (0.20.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/pg-0.20.0.gemspec
    > Using slop 3.6.0
    > 0:  slop (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/slop-3.6.0.gemspec
    > Using puma 3.9.1
    > 0:  puma (3.9.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/puma-3.9.1.gemspec
    > Using r2 0.2.6
    > 0:  r2 (0.2.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/r2-0.2.6.gemspec
    > Using raindrops 0.18.0
    > 0:  raindrops (0.18.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/raindrops-0.18.0.gemspec
    > Using trollop 2.1.2
    > 0:  trollop (2.1.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/trollop-2.1.2.gemspec
    > Using rinku 2.0.2
    > 0:  rinku (2.0.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rinku-2.0.2.gemspec
    > Using rspec-support 3.6.0
    > 0:  rspec-support (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-support-3.6.0.gemspec
    > Using rtlit 0.0.5
    > 0:  rtlit (0.0.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rtlit-0.0.5.gemspec
    > Using sass 3.4.24
    > 0:  sass (3.4.24) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sass-3.4.24.gemspec
    > Using shoulda-context 1.2.2
    > 0:  shoulda-context (1.2.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/shoulda-context-1.2.2.gemspec
    > Using simple-rss 1.3.1
    > 0:  simple-rss (1.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/simple-rss-1.3.1.gemspec
    > Using spork 1.0.0rc4
    > 0:  spork (1.0.0rc4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/spork-1.0.0rc4.gemspec
    > Using stackprof 0.2.10
    > 0:  stackprof (0.2.10) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/stackprof-0.2.10.gemspec
    > Using tilt 2.0.7
    > 0:  tilt (2.0.7) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/tilt-2.0.7.gemspec
    > Using unf_ext 0.0.7.4
    > 0:  unf_ext (0.0.7.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/unf_ext-0.0.7.4.gemspec
    > Using rake-compiler 1.0.4
    > 0:  rake-compiler (1.0.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rake-compiler-1.0.4.gemspec
    > Using tzinfo 1.2.3
    > 0:  tzinfo (1.2.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/tzinfo-1.2.3.gemspec
    > Using nokogiri 1.8.0
    > 0:  nokogiri (1.8.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/nokogiri-1.8.0.gemspec
    > Using rack-test 0.6.3
    > 0:  rack-test (0.6.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rack-test-0.6.3.gemspec
    > Using message_bus 2.0.2
    > 0:  message_bus (2.0.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/message_bus-2.0.2.gemspec
    > Using rack-mini-profiler 0.10.5
    > 0:  rack-mini-profiler (0.10.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rack-mini-profiler-0.10.5.gemspec
    > Using rack-protection 1.5.3
    > 0:  rack-protection (1.5.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rack-protection-1.5.3.gemspec
> Fetching mime-types 3.1
> Installing mime-types 3.1
> 0:  mime-types (3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mime-types-3.1.gemspec
> Using addressable 2.5.1
> 0:  addressable (2.5.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/addressable-2.5.1.gemspec
> Using ruby-ll 2.1.2
> 0:  ruby-ll (2.1.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ruby-ll-2.1.2.gemspec
> Using aws-sdk-core 2.5.3
> 0:  aws-sdk-core (2.5.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/aws-sdk-core-2.5.3.gemspec
> Using ember-data-source 2.2.1
> 0:  ember-data-source (2.2.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ember-data-source-2.2.1.gemspec
> Using barber 0.11.2
> 0:  barber (0.11.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/barber-0.11.2.gemspec
> Using uglifier 3.2.0
> 0:  uglifier (3.2.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/uglifier-3.2.0.gemspec
> Using bootsnap 1.0.0
> 0:  bootsnap (1.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/bootsnap-1.0.0.gemspec
> Using sprockets 3.7.1
> 0:  sprockets (3.7.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sprockets-3.7.1.gemspec
> Using crack 0.4.3
> 0:  crack (0.4.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/crack-0.4.3.gemspec
> Using discourse_image_optim 0.24.5
> 0:  discourse_image_optim (0.24.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/discourse_image_optim-0.24.5.gemspec
> Using faraday 0.11.0
> 0:  faraday (0.11.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/faraday-0.11.0.gemspec
> Using rb-inotify 0.9.8
> 0:  rb-inotify (0.9.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rb-inotify-0.9.8.gemspec
> Using omniauth 1.6.1
> 0:  omniauth (1.6.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-1.6.1.gemspec
> Using mini_racer 0.1.11
> 0:  mini_racer (0.1.11) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mini_racer-0.1.11.gemspec
> Using mocha 1.2.1
> 0:  mocha (1.2.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mocha-1.2.1.gemspec
> Using rack-openid 1.3.1
> 0:  rack-openid (1.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rack-openid-1.3.1.gemspec
> Using openid-redis-store 0.0.2
> 0:  openid-redis-store (0.0.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/openid-redis-store-0.0.2.gemspec
> Using redis-namespace 1.5.3
> 0:  redis-namespace (1.5.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/redis-namespace-1.5.3.gemspec
> Using pry 0.10.4
> 0:  pry (0.10.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/pry-0.10.4.gemspec
> Using unicorn 5.3.0
> 0:  unicorn (5.3.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/unicorn-5.3.0.gemspec
> Using rbtrace 0.4.8
> 0:  rbtrace (0.4.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rbtrace-0.4.8.gemspec
> Using rspec-core 3.6.0
> 0:  rspec-core (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-core-3.6.0.gemspec
> Using rspec-expectations 3.6.0
> 0:  rspec-expectations (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-expectations-3.6.0.gemspec
> Using rspec-mocks 3.6.0
> 0:  rspec-mocks (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-mocks-3.6.0.gemspec
> Using sassc 1.11.2
> 0:  sassc (1.11.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sassc-1.11.2.gemspec
> Using unf 0.1.4
> 0:  unf (0.1.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/unf-0.1.4.gemspec
> Using fast_xor 1.1.3
> 0:  fast_xor (1.1.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/fast_xor-1.1.3.gemspec
> Fetching activesupport 4.2.9
> Installing activesupport 4.2.9
> 0:  activesupport (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/activesupport-4.2.9.gemspec
> Using loofah 2.0.3
> 0:  loofah (2.0.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/loofah-2.0.3.gemspec
> Using nokogumbo 1.4.13
> 0:  nokogumbo (1.4.13) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/nokogumbo-1.4.13.gemspec
> Using ruby-readability 0.7.0
> 0:  ruby-readability (0.7.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ruby-readability-0.7.0.gemspec
> Using sidekiq 5.0.3
> 0:  sidekiq (5.0.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sidekiq-5.0.3.gemspec
> Using mail 2.6.6
> 0:  mail (2.6.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/mail-2.6.6.gemspec
> Using oga 2.10
> 0:  oga (2.10) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/oga-2.10.gemspec
> Using aws-sdk-resources 2.5.3
> 0:  aws-sdk-resources (2.5.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/aws-sdk-resources-2.5.3.gemspec
> Using ember-handlebars-template 0.7.5
> 0:  ember-handlebars-template (0.7.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ember-handlebars-template-0.7.5.gemspec
> Using webmock 3.0.1
> 0:  webmock (3.0.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/webmock-3.0.1.gemspec
> Using oauth2 1.3.1
> 0:  oauth2 (1.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/oauth2-1.3.1.gemspec
> Using listen 3.1.5
> 0:  listen (3.1.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/listen-3.1.5.gemspec
> Using omniauth-oauth 1.1.0
> 0:  omniauth-oauth (1.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-oauth-1.1.0.gemspec
> Using omniauth-openid 1.0.1
> 0:  omniauth-openid (1.0.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-openid-1.0.1.gemspec
> Using pry-nav 0.2.4
> 0:  pry-nav (0.2.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/pry-nav-0.2.4.gemspec
> Using pry-rails 0.3.4
> 0:  pry-rails (0.3.4) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/pry-rails-0.3.4.gemspec
> Using rspec 3.6.0
> 0:  rspec (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-3.6.0.gemspec
> Using rails-deprecated_sanitizer 1.0.3
> 0:  rails-deprecated_sanitizer (1.0.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rails-deprecated_sanitizer-1.0.3.gemspec
> Using globalid 0.4.0
> 0:  globalid (0.4.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/globalid-0.4.0.gemspec
> Fetching activemodel 4.2.9
> Installing activemodel 4.2.9
> 0:  activemodel (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/activemodel-4.2.9.gemspec
> Using shoulda-matchers 2.8.0
> 0:  shoulda-matchers (2.8.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/shoulda-matchers-2.8.0.gemspec
> Using rails-html-sanitizer 1.0.3
> 0:  rails-html-sanitizer (1.0.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rails-html-sanitizer-1.0.3.gemspec
> Using sanitize 4.5.0
> 0:  sanitize (4.5.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sanitize-4.5.0.gemspec
> Using aws-sdk 2.5.3
> 0:  aws-sdk (2.5.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/aws-sdk-2.5.3.gemspec
> Using omniauth-oauth2 1.4.0
> 0:  omniauth-oauth2 (1.4.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-oauth2-1.4.0.gemspec
> Using omniauth-twitter 1.3.0
> 0:  omniauth-twitter (1.3.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-twitter-1.3.0.gemspec
> Using rspec-html-matchers 0.9.1
> 0:  rspec-html-matchers (0.9.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-html-matchers-0.9.1.gemspec
> Using rails-dom-testing 1.0.8
> 0:  rails-dom-testing (1.0.8) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rails-dom-testing-1.0.8.gemspec
> Fetching activejob 4.2.9
> Installing activejob 4.2.9
> 0:  activejob (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/activejob-4.2.9.gemspec
> Using active_model_serializers 0.8.3
> 0:  active_model_serializers (0.8.3) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/active_model_serializers-0.8.3.gemspec
> Fetching activerecord 4.2.9
> Installing activerecord 4.2.9
> 0:  activerecord (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/activerecord-4.2.9.gemspec
> Using shoulda 3.5.0
> 0:  shoulda (3.5.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/shoulda-3.5.0.gemspec
> Fetching onebox 1.8.17
> Installing onebox 1.8.17
> 0:  onebox (1.8.17) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/onebox-1.8.17.gemspec
> Using omniauth-facebook 4.0.0
> 0:  omniauth-facebook (4.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-facebook-4.0.0.gemspec
> Fetching omniauth-github 1.3.0
> Installing omniauth-github 1.3.0
> 0:  omniauth-github (1.3.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-github-1.3.0.gemspec
> Using omniauth-google-oauth2 0.3.1
> 0:  omniauth-google-oauth2 (0.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-google-oauth2-0.3.1.gemspec
> Using omniauth-instagram 1.0.2
> 0:  omniauth-instagram (1.0.2) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/omniauth-instagram-1.0.2.gemspec
> Fetching actionview 4.2.9
> Installing actionview 4.2.9
> 0:  actionview (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/actionview-4.2.9.gemspec
> Using seed-fu 2.3.6
> 0:  seed-fu (2.3.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/seed-fu-2.3.6.gemspec
> Using test_after_commit 1.1.0
> 0:  test_after_commit (1.1.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/test_after_commit-1.1.0.gemspec
> Fetching actionpack 4.2.9
> Installing actionpack 4.2.9
> 0:  actionpack (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/actionpack-4.2.9.gemspec
> Fetching actionmailer 4.2.9
> Installing actionmailer 4.2.9
> 0:  actionmailer (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/actionmailer-4.2.9.gemspec
> Fetching railties 4.2.9
> Installing railties 4.2.9
> 0:  railties (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/railties-4.2.9.gemspec
> Using sprockets-rails 3.2.0
> 0:  sprockets-rails (3.2.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/sprockets-rails-3.2.0.gemspec
> Using discourse-qunit-rails 0.0.9
> 0:  discourse-qunit-rails (0.0.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/discourse-qunit-rails-0.0.9.gemspec
> Using jquery-rails 4.3.1
> 0:  jquery-rails (4.3.1) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/jquery-rails-4.3.1.gemspec
> Using rspec-rails 3.6.0
> 0:  rspec-rails (3.6.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rspec-rails-3.6.0.gemspec
> Fetching rails 4.2.9
> Installing rails 4.2.9
> 0:  rails (4.2.9) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rails-4.2.9.gemspec
> Using ember-rails 0.18.5
> 0:  ember-rails (0.18.5) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/ember-rails-0.18.5.gemspec
> Using rails_multisite 1.0.6
> 0:  rails_multisite (1.0.6) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/rails_multisite-1.0.6.gemspec
> Using spork-rails 4.0.0
> 0:  spork-rails (4.0.0) from /var/www/discourse/vendor/bundle/ruby/2.3.0/specifications/spork-rails-4.0.0.gemspec
> Bundle complete! 96 Gemfile dependencies, 173 gems now installed.
> Gems in the group development were not installed.
> Bundled gems are installed into ./vendor/bundle.
> 
> I, [2017-08-08T23:52:47.811715 #16]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
> rake aborted!
> NoMethodError: undefined method `[]' for nil:NilClass
> /var/www/discourse/plugins/custom-html-widget/plugin.rb:9:in `block (2 levels) in activate!'
> /var/www/discourse/plugins/custom-html-widget/plugin.rb:8:in `class_eval'
> /var/www/discourse/plugins/custom-html-widget/plugin.rb:8:in `block in activate!'
> /var/www/discourse/lib/plugin/instance.rb:208:in `block in notify_after_initialize'
> /var/www/discourse/lib/plugin/instance.rb:206:in `each'
> /var/www/discourse/lib/plugin/instance.rb:206:in `notify_after_initialize'
> /var/www/discourse/config/application.rb:209:in `block (2 levels) in <class:Application>'
> /var/www/discourse/config/application.rb:209:in `each'
> /var/www/discourse/config/application.rb:209:in `block in <class:Application>'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/lazy_load_hooks.rb:45:in `block in run_load_hooks'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/lazy_load_hooks.rb:44:in `each'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/lazy_load_hooks.rb:44:in `run_load_hooks'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/application/finisher.rb:62:in `block in <module:Finisher>'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `instance_exec'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `run'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/initializable.rb:55:in `block in run_initializers'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/initializable.rb:54:in `run_initializers'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/application.rb:352:in `initialize!'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/railtie.rb:194:in `public_send'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/railtie.rb:194:in `method_missing'
> /var/www/discourse/config/environment.rb:5:in `<top (required)>'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:274:in `require'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:274:in `block in require'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:240:in `load_dependency'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:274:in `require'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/application.rb:328:in `require_environment!'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/railties-4.2.9/lib/rails/application.rb:457:in `block in run_tasks_blocks'
> /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
> /usr/local/bin/bundle:22:in `load'
> /usr/local/bin/bundle:22:in `<main>'
> Tasks: TOP => db:migrate => environment
> (See full trace by running task with --trace)
> I, [2017-08-08T23:53:01.368713 #16]  INFO -- : gem install hkdf -v 0.2.0 -i /var/www/discourse/plugins/discourse-push-notifications/gems/2.3.4 --no-document --ignore-dependencies
> Successfully installed hkdf-0.2.0
> 1 gem installed
> gem install webpush -v 0.3.1 -i /var/www/discourse/plugins/discourse-push-notifications/gems/2.3.4 --no-document --ignore-dependencies
> Successfully installed webpush-0.3.1
> 1 gem installed
> 
> I, [2017-08-08T23:53:01.369770 #16]  INFO -- : Terminating async processes
> I, [2017-08-08T23:53:01.370051 #16]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main pid: 45
> 2017-08-08 23:53:01.370 UTC [45] LOG:  received fast shutdown request
> 2017-08-08 23:53:01.370 UTC [45] LOG:  aborting any active transactions
> I, [2017-08-08T23:53:01.370744 #16]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 158
> 158:signal-handler (1502236381) Received SIGTERM scheduling shutdown...
> 2017-08-08 23:53:01.371 UTC [52] LOG:  autovacuum launcher shutting down
> 2017-08-08 23:53:01.374 UTC [49] LOG:  shutting down
> 2017-08-08 23:53:01.394 UTC [49] LOG:  database system is shut down
> 158:M 08 Aug 23:53:01.428 # User requested shutdown...
> 158:M 08 Aug 23:53:01.428 * Saving the final RDB snapshot before exiting.
> 158:M 08 Aug 23:53:01.966 * DB saved on disk
> 158:M 08 Aug 23:53:01.966 # Redis is now ready to exit, bye bye...
> 
> 
> FAILED
> --------------------
> Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 535 exit 1>
> Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
> exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
> a1fbefc6190fcf4da9f1e7ce75765223d3637387fac1576f4fd82cd46238c47c
> ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one
> root@discourse-2gb-nyc3-01:/var/discourse#

(cpradio) #20

Can you remove the custom-html-widget from your app.yml and try the rebuild again?