Failed upgrade from 2.9.0.beta14 to 3.1.0.beta

installation
1

I upgrade every month, always on a few development systems first, then production.

During this month’s cycle, the upgrade failed on the 1st development system. docker_manager upgraded fine but the discourse upgrade failed with:

Sorry, there was an error upgrading Discourse. Please check the logs below.

The Discourse app is completely down (it’s displaying the “Oops” error 500 page).

Here’s what the on-screen log contains:

********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 50
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Waiting for Unicorn to reload..........
Waiting for Unicorn to reload...........
Stopping 7 Unicorn worker(s), to free up memory
Stopping job queue to reclaim memory, master pid is 4069
$ cd /var/www/discourse && git fetch --tags --force && git reset --hard HEAD@{upstream}
From https://github.com/discourse/discourse
 t [tag update]            latest-release -> latest-release
 * [new tag]               v2.8.14        -> v2.8.14
 * [new tag]               v3.0.0         -> v3.0.0
HEAD is now at 666536cbd1 DEV: Prefer \A and \z over ^ and $ in regexes (#19936)
$ bundle install --deployment --jobs 4 --without test development
[DEPRECATED] The `--deployment` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local deployment 'true'`, and stop using this flag
[DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local without 'test development'`, and stop using this flag
Bundler 2.3.13 is running, but your lockfile was generated with 2.4.1. Installing Bundler 2.4.1 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.4.1
Installing bundler 2.4.1
[DEPRECATED] The `--deployment` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local deployment 'true'`, and stop using this flag
[DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local without 'test development'`, and stop using this flag
Fetching gem metadata from https://rubygems.org/.........
Fetching https://github.com/rails/sprockets
web-push-3.0.0 requires ruby version >= 3.0, which is incompatible with the
current version, 2.7.6
Docker Manager: FAILED TO UPGRADE
#<RuntimeError: RuntimeError>
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:209:in `run'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:93:in `upgrade'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:19:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:20:in `block in fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `load'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command/base.rb:87:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command.rb:48:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands.rb:18:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
bin/rails:18:in `<main>'
Spinning up 7 Unicorn worker(s) that were stopped initially

I’ve never had an upgrade error before so I’m unclear what to do next. And if this had happened in production, what would I do?

2 Likes
2

I suggest you check what version of operating system (distribution) you are running: it could be that it’s an old version, doesn’t have ruby 3, and that’s the problem.

Edit: hang on, I’m not sure if I might be confused. For production, everything happens within a Docker container which should have all the right versions of everything. Do you have some other kind of setup?

2 Likes
3

FWIW, the dev boxes are Ubuntu 20.04 LTS. I believe prod is 22.04 LTS.

However, I’ve never installed Ruby on the OS. I think all of that happens automagically in the docker container.

Correction: Prod is also on 20.04 LTS.

1 Like
4

Yes, sorry, I think I was confused. It’s not uncommon to need to do a command-line upgrade - have you tried that?

cd /var/discourse
git pull
./launcher rebuild app
6 Likes
5

If those are development installs they won’t be representative of a production docker install. Your dev environment has an old version of ruby (2.7), whereas Discourse recently bumped up to 3.1 in the image.

6

Thanks! Rebuilding the app seems to have worked. I’m trying this on another dev system before patching production.

7

To be clear, by “development” I simply mean they are just like production but on internal development systems.

1 Like
8

Did an upgrade via bash resolve things?

1 Like
9

Yep! It worked on both dev systems and prod too. Thanks!

3 Likes
10

This is happen to me to in my production.

From 2.9.0.beta14 (2b9fa41a6e) to 3.1.0.beta1

Fetching gem metadata from https://rubygems.org/.........
Fetching https://github.com/rails/sprockets
web-push-3.0.0 requires ruby version >= 3.0, which is incompatible with the
current version, 2.7.6
Docker Manager: FAILED TO UPGRADE
#<RuntimeError: RuntimeError>
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:209:in `run'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:93:in `upgrade'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:19:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:20:in `block in fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `load'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command/base.rb:87:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command.rb:48:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands.rb:18:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
bin/rails:18:in `<main>'
Spinning up 1 Unicorn worker(s) that were stopped initially

Solved with :point_up:

cd /var/discourse
git pull
./launcher rebuild app

thanks

11

Same issue here, upgrade failed and nuked the forum to 500 errors. I’m honestly rather surprised this issue persisted for 3 days so far.

Anyway I’m running a CLI rebuild now which will hopefully work. If not, I took a VM snapshot first because my momma didn’t raise no fool.

From this post it looks like they upgraded ruby to 3.0 which completely breaks web upgrades. Unfortunately the software still lets users try it, and then you get to see how quickly your host can serve up HTTP 500 errors.

Edit: The CLI rebuild did indeed work, phew!

Relevant errors:

Fetching gem metadata from https://rubygems.org/.........
Fetching https://github.com/rails/sprockets
web-push-3.0.0 requires ruby version >= 3.0, which is incompatible with the
current version, 2.7.6
Docker Manager: FAILED TO UPGRADE
#<RuntimeError: RuntimeError>
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:209:in `run'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:93:in `upgrade'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:19:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:20:in `block in fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.3.1/lib/active_support/fork_tracker.rb:18:in `fork'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `load'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands/runner/runner_command.rb:43:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command/base.rb:87:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/command.rb:48:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/railties-7.0.3.1/lib/rails/commands.rb:18:in `<main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
bin/rails:18:in `<main>'
2 Likes
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.