Upgrade failing with FAILED TO BOOTSTRAP

[burke]

When trying to upgrade our Discourse instance, it required a git pull today, but the ./launcher rebuild is failing with FAILED TO BOOTSTRAP and I’m not yet seeing where the failure is coming from.

We’re running Discourse on Ubuntu 18.04. Everything has been going fine up until today’s upgrade.

Our app.yml is called web.yml, so I ran:

# git pull
# ./launcher rebuild web

The tail of the output looks like this:

I, [2022-02-18T19:25:46.155360 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
Discourse requires Redis 6.2.0 or up
I, [2022-02-18T19:25:55.644442 #1]  INFO -- : gem install sawyer -v 0.8.2 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install
Successfully installed sawyer-0.8.2
1 gem installed
gem install octokit -v 4.21.0 -i /var/www/discourse/plugins/discourse-github/gems/2.7.5 --no-document --ignore-dependencies --no-user-install
Successfully installed octokit-4.21.0
1 gem installed
 
 
 
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 1121 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
85459e34ac2c6275dd1700de2c469124a9fded84800b8c6b4686c1c6b66824e2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Full output here

These are our current plugins:

discourse/docker_manager.git
discourse/discourse-github.git
discourse/discourse-solved.git
discourse/discourse-data-explorer.git
discourse/discourse-akismet.git
discourse/discourse-spoiler-alert.git
cpradio/discourse-plugin-replygif.git
discourse/discourse-push-notifications.git
discourse/discourse-chat-integration

When I scan the launcher output, I’m not seeing the error that’s causing the failure. Any suggestions on what might be causing the failure?

I tried reverting the previous commit to see if I could successfully rebuild our web container without the latest change, but it still fails with a failure to bootstrap.

Update:

We were using a separate containers setup, which is apparently not a good idea, and our data container was ancient, running Redis v5.0.5 (Discourse currently requires 6.2.0+). So, we rebuilt our data container and then web containers successfully. After this experience, we’ll probably abandon the multiple container approach going forward.

Thanks to @pfaffman for the incredibly helpful links!

[pfaffman (Jay Pfaffman)]

@burke

Discourse requires Redis 6.2.0 or up

What Redis are you using?

See also Skip Redis Version Check

[burke]

Ah. I see. We’re running discourse_docker with linked web and data containers, so Redis is running within our data container.

Within our data container:

# redis-server --version
Redis server v=5.0.5

So, that explains it. It looks like we need to upgrade our data container as well. That makes me more nervous. When was that requirement added? Maybe I could revert to a previous version of the web container until my reinforcements (in Australia) are awake & available.

[burke]

Wow. Looking through git history it looks like the Redis version hasn’t been in 5.x range for two years! But I know we’ve done some successful upgrades along the way (including git pulls).

Based on Skip Redis Version Check, it looks like they recently started getting enforced. I’ve already dug a big enough hole for a Friday and don’t want to try upgrading our data container without some help… so, I’ll try reverting to an even earlier version of docker_discourse to see if I can get to back to a state where the Redis requirement isn’t enforced for now (until we can get our data container updated).

[Falco]

@burke

I’ll try reverting to an even earlier version of docker_discourse

That won’t work, as the check comes from the main repo. Leaving the data constainers without updates for multiple years is not great, as we shipped performance and security fixes many times on that period.

Yet another reason I do not recomment the separate containers setup.

[pfaffman (Jay Pfaffman)]

You need to rebuild the data container. See also PostgreSQL 13 update

[burke]

Thanks. Waiting on Postgres upgrade as we speak.

And thanks for the link to the Postgres 13 update tips. Very helpful!