Rebuild fails from theme error, site won't start

[FoohonPie]

Hello, this morning my Discourse failed for me mid-usage, and rebuilding has me stuck. Initially I had the “Oops!” page, but after my first rebuild it won’t start at all.

• I’ve removed all plugins, official or otherwise, leaving only docker_manager.git
• I’ve git pulled from within /var/discourse and am Already up-to-date.
• I’ve tried ./launcher cleanup, but ./launcher rebuild app still fails

The error shown is:

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 17092 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
f3a13e3833fdc05123d7232621e11b0a0abdcfadad416474f12ee87f319e0f1e
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Going up, I see a rake aborted! error:

Log

Start compiling CSS: 2019-02-08 20:21:39 UTC
Compiling css for default 2019-02-08 20:21:39 UTC
precompile target: desktop Dark
rake aborted!
ArgumentError: comparison of Integer with nil failed
/var/www/discourse/app/models/theme.rb:132:in `sort!'
/var/www/discourse/app/models/theme.rb:132:in `block in transform_ids'
/var/www/discourse/app/models/theme.rb:88:in `get_set_cache'
/var/www/discourse/app/models/theme.rb:123:in `transform_ids'
/var/www/discourse/app/models/theme.rb:332:in `all_theme_variables'
/var/www/discourse/lib/stylesheet/importer.rb:43:in `block in <class:Importer>'
/var/www/discourse/lib/stylesheet/importer.rb:151:in `instance_eval'
/var/www/discourse/lib/stylesheet/importer.rb:151:in `imports'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sassc-1.11.4/lib/sassc/import_handler.rb:29:in `block in import_function'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sassc-1.11.4/lib/sassc/engine.rb:40:in `compile_data_context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sassc-1.11.4/lib/sassc/engine.rb:40:in `render'
/var/www/discourse/lib/stylesheet/compiler.rb:38:in `compile'
/var/www/discourse/lib/stylesheet/compiler.rb:20:in `compile_asset'
/var/www/discourse/lib/stylesheet/manager.rb:162:in `compile'
/var/www/discourse/lib/stylesheet/manager.rb:96:in `block (2 levels) in precompile_css'
/var/www/discourse/lib/stylesheet/manager.rb:89:in `each'
/var/www/discourse/lib/stylesheet/manager.rb:89:in `block in precompile_css'
/var/www/discourse/lib/stylesheet/manager.rb:88:in `each'
/var/www/discourse/lib/stylesheet/manager.rb:88:in `precompile_css'
/var/www/discourse/lib/tasks/assets.rake:60:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/assets.rake:53:in `block in <top (required)>'
/var/www/discourse/lib/tasks/assets.rake:235:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'

I gather there may be a theme and/or component error, but I cannot enter safe mode to disable anything because my site won’t start at all.

What do you recommend to do next in this scenario?

Thank you.

[david (David Taylor)]

Can you try running

./launcher start app

That should bring the site back online (on the old version). Once it’s back up, please can you run

./launcher enter app
rails c

SiteSetting.default_theme_id

[FoohonPie]

Thanks @david,

./launcher start app fails to bring the site back up:

Unable to find image 'local_discourse/app:latest' locally
/usr/bin/docker: Error response from daemon: repository local_discourse/app not found: does not exist or no pull access.
See '/usr/bin/docker run --help'.

docker ps returns empty, there’s no app container to enter.

[david (David Taylor)]

Ah, it’s possible that the old container was cleaned up when you ran launcher cleanup (since it wasn’t running).

Ok, can you try editing your app.yml file, and add this under the env section. Make sure the indentation is the same as all the other variables in that section.

DONT_PRECOMPILE_CSS: 1

Then try rebuilding again.

[FoohonPie]

Perfect, that’s just what I needed.

Thanks, I appreciate it!

[david (David Taylor)]

Hold up! You don’t want to leave it like that forever - we precompile CSS for a reason

Now that the site is up, can you try running the commands I listed above:

./launcher enter app
rails c

SiteSetting.default_theme_id

That should give you a number. If so, try

Theme.find(SiteSetting.default_theme_id).name

[FoohonPie]

Oh yeah I know lol. I used it to get back into safe-mode, adjusted what I needed, and then removed it. All good, I ended up not needing to enter the app or rails console.

Thanks for checking though!

[david (David Taylor)]

Did you figure out what caused the issue in the first place? If possible I’d like to find out the root cause so that we can prevent it happening to anyone in the future - broken rebuilds are pretty annoying!

[FoohonPie]

I’m not sure what caused my site to break mid-use, but I do believe the rebuild breaking has something to do a recurring issue with an unofficial plugin.

The details are here if you’re curious, but it doesn’t seem to be an issue with core or any official plugins as far as I can tell.

[david (David Taylor)]

Thanks for the information.

My suspicion is that somehow the “default” theme got deleted, which would explain the site breaking, and then the rebuild failing. I’m not exactly sure how that could happen, but I’ll double check our safeguards on that next week (I’m working on this code anyway).