Failure trying to rebuild. Server dead in the water

My server failed again because it ran out of disk space, because the automated backups are still generating more than they should (whole nother story). I deleted the oldest backup, and ran ./launcher rebuild app. Got the following, multiple times:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 450 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
c45fbf7d797e6039ef7ee7cd2d07692df73ce7622861af489b112ec498a93335
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Scrolling up, I see the following stack dump:

www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/base.rb:66:in `find_asset'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/base.rb:73:in `find_all_linked_assets'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:134:in `block in find'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:133:in `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:133:in `find'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:185:in `compile'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-rails-3.1.1/lib/sprockets/rails/task.rb:68:in `block (3 levels) in define'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/rake/sprocketstask.rb:147:in `with_logger'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-rails-3.1.1/lib/sprockets/rails/task.rb:67:in `block (2 levels) in define'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'
MiniRacer::RuntimeError: Error: Assertion Failed: Using `{{controller}}` or any path based on it ('javascripts/connectors/user-custom-preferences/protect-history' @ L3:C28) has been removed in Ember 2.0
JavaScript at new Error (native)
JavaScript at Error.EmberError (<anonymous>:9091:21)
JavaScript at assert (<anonymous>:6024:13)
JavaScript at Object.assert (<anonymous>:8855:34)
JavaScript at assertPath (<anonymous>:16427:22)
JavaScript at assertPaths (<anonymous>:16422:7)
JavaScript at assertHash (<anonymous>:16411:7)
JavaScript at <anonymous>:16397:7
JavaScript at Walker.visit (<anonymous>:23457:7)
JavaScript at ElementNode (<anonymous>:23473:16)
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:167:in `eval_unsafe'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:167:in `block (2 levels) in eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:217:in `timeout'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:166:in `block in eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:63:in `block in with_lock'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:63:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:63:in `with_lock'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/mini_racer-0.1.7/lib/mini_racer.rb:164:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/execjs-2.7.0/lib/execjs/mini_racer_runtime.rb:27:in `block in eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/execjs-2.7.0/lib/execjs/mini_racer_runtime.rb:66:in `translate'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/execjs-2.7.0/lib/execjs/mini_racer_runtime.rb:26:in `eval'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/execjs-2.7.0/lib/execjs/mini_racer_runtime.rb:34:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/barber-0.11.1/lib/barber/precompiler.rb:31:in `compile'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/barber-0.11.1/lib/barber/precompiler.rb:8:in `compile'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/ember-handlebars-template-0.7.4/lib/ember/handlebars/template.rb:166:in `block in precompile_ember_handlebars'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cache.rb:85:in `fetch'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/ember-handlebars-template-0.7.4/lib/ember/handlebars/template.rb:165:in `precompile_ember_handlebars'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/ember-handlebars-template-0.7.4/lib/ember/handlebars/template.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/ember-handlebars-template-0.7.4/lib/ember/handlebars/template.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:75:in `call_processor'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `reverse_each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:75:in `call_processor'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `reverse_each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:134:in `load_from_unloaded'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:60:in `block in load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:44:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/bundle.rb:23:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/utils.rb:196:in `dfs'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/bundle.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:75:in `call_processor'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `reverse_each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/processor_utils.rb:56:in `call_processors'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:134:in `load_from_unloaded'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:60:in `block in load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/loader.rb:44:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:20:in `block in initialize'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/cached_environment.rb:47:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/base.rb:66:in `find_asset'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/base.rb:73:in `find_all_linked_assets'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:134:in `block in find'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:133:in `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:133:in `find'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/sprockets/manifest.rb:185:in `compile'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-rails-3.1.1/lib/sprockets/rails/task.rb:68:in `block (3 levels) in define'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-3.6.3/lib/rake/sprocketstask.rb:147:in `with_logger'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/sprockets-rails-3.1.1/lib/sprockets/rails/task.rb:67:in `block (2 levels) in define'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'

Any words of wisdom? I have some unhappy users.

One of your plugins (user-custom-preferences, perhaps?) is incompatible with the most recent version of Discourse. Remove it (or update it to be compatible) and your rebuild shouldl work just fine.

3 Likes

Thanks for the quick reply! Any suggestions on how I do that in this state?

What is the list of plugins you have defined in your app.yml file?

Do a binary search to find the broken one by commenting out bits.

1 Like

Ah, totally forgot about app.yml. Working with too many different environments. Thanks, Sam, I’ll give that a try.

And, the winner is…

browser_history!

https://github.com/discourse/browser_history/blob/master/plugin.rb

2 Likes

Thanks for your help, Sam and Matt!