Rebuilding on 2.8.13 stable (not upgrading, already was running without issue on 2.8.13) is failing for me.
Normally I can run ./launcher start app to get the previous container back up if the rebuild fails (though I haven’t needed to use this in quite a long time now). This isn’t working either, it ‘works’ on command line like normal, but the site remains offline and inaccessible.
Nothing in the config was changed on server B, tried rebuilding with the unchanged config as well as with docker manager only and no plugins at all. On server A I tried just adding a simple plugin (that is already running on other servers fine using 2.8.13), which is why I was rebuilding in the first place - also tried on that server with no plugins as well as with docker manager only without success.
The existing plugins prior to upgrade were all up to date except for docker-manager, according to the web panel. Tested removing pretty much everything mildly custom from the app.yml also.
These are the templates being used:
templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
- "templates/cloudflare.template.yml"
Ended up just restoring a vm snapshot. Experienced the same issue on two different servers (the only two I tried).
I tried again on the restored VM for the sake of getting relevant parts of the logs, please find them below:
From https://github.com/discourse/discourse_docker
e0f27d1..0595fc2 main -> origin/main
+ 4fae4a0...2e142d4 ruby-3.2 -> origin/ruby-3.2 (forced update)
bdc1b59..4ffccf2 yarn-timeout -> origin/yarn-timeout
Updating Launcher...
Updating e0f27d1..0595fc2
Fast-forward
image/base/VERSION | 1 -
image/base/install-imagemagick | 4 ++--
image/base/install-jemalloc | 10 +++++-----
image/base/install-nginx | 4 ++--
image/base/install-oxipng | 4 ++--
image/base/install-redis | 4 ++--
image/base/install-ruby | 7 ++++++-
image/base/install-rust | 30 ++++++++++++++++++++++++++++++
image/base/slim.Dockerfile | 12 +++++++++---
launcher | 2 +-
10 files changed, 59 insertions(+), 19 deletions(-)
delete mode 100644 image/base/VERSION
create mode 100755 image/base/install-rust
Launcher updated, restarting...
x86_64 arch detected.
Post-install message from image_optim:
Rails image assets optimization is extracted into image_optim_rails gem
You can safely remove `config.assets.image_optim = false` if you are not going to use that gem
I, [2022-12-28T05:28:46.999225 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2022-12-28T05:28:51.255157 #1] INFO -- : docker_manager is already at latest compatible version
I, [2022-12-28T05:28:51.256526 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
LoadError: cannot load such file -- net/pop
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:93:in `register'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:44:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `block in require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:299:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `require'
/var/www/discourse/lib/validators/pop3_polling_enabled_setting_validator.rb:3:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `block in require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:299:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:424:in `block in require_or_load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:39:in `block in load_interlock'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:13:in `loading'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:39:in `load_interlock'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:402:in `require_or_load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:558:in `load_missing_constant'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:213:in `const_missing'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/inflector/methods.rb:274:in `const_get'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/inflector/methods.rb:274:in `constantize'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/core_ext/string/inflections.rb:74:in `constantize'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/core_ext/object/try.rb:15:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/core_ext/object/try.rb:15:in `try'
/var/www/discourse/lib/site_settings/type_supervisor.rb:114:in `load_setting'
/var/www/discourse/lib/site_setting_extension.rb:170:in `block in setting'
/var/www/discourse/lib/site_setting_extension.rb:127:in `synchronize'
/var/www/discourse/lib/site_setting_extension.rb:127:in `setting'
/var/www/discourse/app/models/site_setting.rb:12:in `block in load_settings'
/var/www/discourse/lib/site_settings/yaml_loader.rb:22:in `block (2 levels) in load'
/var/www/discourse/lib/site_settings/yaml_loader.rb:13:in `each'
/var/www/discourse/lib/site_settings/yaml_loader.rb:13:in `block in load'
/var/www/discourse/lib/site_settings/yaml_loader.rb:12:in `each_key'
/var/www/discourse/lib/site_settings/yaml_loader.rb:12:in `load'
/var/www/discourse/app/models/site_setting.rb:11:in `load_settings'
/var/www/discourse/app/models/site_setting.rb:16:in `<class:SiteSetting>'
/var/www/discourse/app/models/site_setting.rb:3:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `block in require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:299:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:424:in `block in require_or_load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:39:in `block in load_interlock'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:13:in `loading'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:39:in `load_interlock'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:402:in `require_or_load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:558:in `load_missing_constant'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:213:in `const_missing'
/var/www/discourse/config/initializers/004-message_bus.rb:122:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in `load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in `load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:326:in `block in load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:299:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:326:in `load'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/engine.rb:681:in `block in load_config_initializer'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/notifications.rb:205:in `instrument'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/engine.rb:680:in `load_config_initializer'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/engine.rb:634:in `block (2 levels) in <class:Engine>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/engine.rb:633:in `each'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/engine.rb:633:in `block in <class:Engine>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `run'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `tsort_each_child'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/application.rb:391:in `initialize!'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `method_missing'
/var/www/discourse/config/environment.rb:7:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:88:in `register'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.9.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `block in require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:299:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies.rb:332:in `require'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/application.rb:367:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/railties-6.1.6.1/lib/rails/application.rb:533:in `block in run_tasks_blocks'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate => db:load_config => environment
(See full trace by running task with --trace)
I, [2022-12-28T05:28:55.997202 #1] INFO -- :
I, [2022-12-28T05:28:56.008251 #1] INFO -- : Terminating async processes
I, [2022-12-28T05:28:56.009774 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
I, [2022-12-28T05:28:56.010776 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
2022-12-28 05:28:56.010 UTC [41] LOG: received fast shutdown request
2022-12-28 05:28:56.020 UTC [41] LOG: aborting any active transactions
102:signal-handler (1672205336) Received SIGTERM scheduling shutdown...
102:M 28 Dec 2022 05:28:56.027 # User requested shutdown...
102:M 28 Dec 2022 05:28:56.029 * Saving the final RDB snapshot before exiting.
2022-12-28 05:28:56.034 UTC [41] LOG: background worker "logical replication launcher" (PID 50) exited with exit code 1
2022-12-28 05:28:56.039 UTC [45] LOG: shutting down
2022-12-28 05:28:56.083 UTC [41] LOG: database system is shut down
102:M 28 Dec 2022 05:28:56.242 * DB saved on disk
102:M 28 Dec 2022 05:28:56.243 # Redis is now ready to exit, bye bye...
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 1797 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.1.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.