Updating Discourse Fails

[arjunaskykok (Arjuna)]

I tried to update Discourse to the latest version today. It failed. Here’s the log:

********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
Cycling Unicorn, to free up memory
Restarting unicorn pid: 548
Waiting for Unicorn to reload.
Waiting for Unicorn to reload..
Waiting for Unicorn to reload...
Waiting for Unicorn to reload....
Waiting for Unicorn to reload.....
Waiting for Unicorn to reload......
Waiting for Unicorn to reload.......
Waiting for Unicorn to reload........
Waiting for Unicorn to reload.........
Stopping 3 Unicorn worker(s), to free up memory
Stopping job queue to reclaim memory, master pid is 2943724
$ cd /var/www/discourse && git fetch --tags --prune-tags --prune --force
$ cd /var/www/discourse && git reset --hard HEAD@{upstream}
HEAD is now at be4f1e335 PERF: Cache `hostname` in `DiscourseLogstashLogger` (#27442)
$ bundle install --retry 3 --jobs 4
Bundle complete! 141 Gemfile dependencies, 174 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/bundle`
1 installed gem you directly depend on is looking for funding.
  Run `bundle fund` for details
$ yarn install
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
[3/5] Fetching packages...
error ember-template-lint@6.0.0: The engine "node" is incompatible with this module. Expected version "^18.18.0 || ^20.9.0 || >=21.1.0". Got "18.16.0"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Docker Manager: FAILED TO UPGRADE
#<RuntimeError: RuntimeError>
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:210:in `run'
/var/www/discourse/plugins/docker_manager/lib/docker_manager/upgrader.rb:101:in `upgrade'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:19:in `block in <main>'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `fork'
/var/www/discourse/plugins/docker_manager/scripts/docker_manager_upgrade.rb:6:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.4/lib/rails/commands/runner/runner_command.rb:43:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.4/lib/rails/commands/runner/runner_command.rb:43:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.4/lib/rails/command/base.rb:87:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.4/lib/rails/command.rb:48:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.4/lib/rails/commands.rb:18:in `<main>'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
bin/rails:18:in `<main>'
Spinning up 3 Unicorn worker(s) that were stopped initially

Any idea?

[pfaffman (Jay Pfaffman)]

You should do a command line upgrade.

./launcher rebuild app

[arjunaskykok (Arjuna)]

Hi Jay,

I’ve done that. I also tried to do ./launcher start app.

But I got a different error. This time, it’s a bad gateway error from Nginx. I noticed that Discourse complained about low storage but I cleaned up some space.

Attached some information.

[pfaffman (Jay Pfaffman)]

Did you do a start or a rebuild after you freed up the space?

The bad gateway is expected for a minute or a few after you start the container. But it may be they you’ve migrated the database and the old container will no longer work.

[arjunaskykok (Arjuna)]

So, every time I do ./launcher rebuild app or ./launcher start app or ./launcher restart app, I’m always greeted with the annoying question about cleaning up the space. I press Y every time.

[arjunaskykok (Arjuna)]

Okay, I cleaned up some space manually. Apparently, there are useless files in the snap directory.

Then I retried again the process ./launcher restart app. This time, the error disappeared.

But I still get the 502 Bad Gateway error.

[pfaffman (Jay Pfaffman)]

If you’re still getting that after the site’ been up for a few minutes then look at /var/discourse/shared/standalone/logs/rails/production.log and also .../logs/var-log/nginx/*

[arjunaskykok (Arjuna)]

So, it’s a PostgreSQL issue. How do I fix this? I mean, we don’t start PostgreSQL server manually right.

The files inside postgres_data are owned by my username in the server and the group is crontab.

[merefield (Robert)]

@arjunaskykok

The files inside postgres_data are owned by my username

Why?

Is this not a standard install?

[arjunaskykok (Arjuna)]

I used the standard install.

Then the update process (via admin dashboard) failed. So I had to ssh into the server and cleaned up some spaces. When I did ./launcher rebuild (or restart) app, the process didn’t throw error, but I got 502 Bad Gateway error.

So, the postgres_data should be owned by whom? Root?

[merefield (Robert)]

This should all be installed as root.

[arjunaskykok (Arjuna)]

I did ./launcher rebuild (or restart) app as root. I did sudo su first. Anyway, I don’t think you could execute launcher with a normal user (Docker would complain).

[arjunaskykok (Arjuna)]

Okay, I finally fixed these problems.

1. I resized the storage to give some room to Discord. I created a snapshot to the Discord VM in GCP and restored it with a bigger storage.
2. I changed the content of the containers/app.yml file:

From

templates:
  - "templates/postgres.template.yml"

to

templates:
  - "templates/postgres.13.template.yml"

Then I did ./launcher rebuild app with no problems anymore.

Thanks!