Discourse rebuild fails after upgrade to 2.0.0 beta7


(Slack-Moehrle) #1

I upgraded and then did ./launcher rebuild app and it fails with

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 415 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
1e4d6fd564f65df325ac6363be6202e0ede69a763f09adc80b83ea3f665a99db
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

If I scroll up, I see:

Bundle complete! 105 Gemfile dependencies, 179 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2018-04-25T02:01:50.334697 #16]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::ConflictError: Unable to activate sidekiq-scheduler-2.0.9, because redis-4.0.1 conflicts with redis (~> 3)
/var/www/discourse/lib/plugin_gem.rb:18:in `load'
/var/www/discourse/lib/plugin/instance.rb:493:in `gem'
/var/www/discourse/plugins/discourse-admin-statistics-digest/plugin.rb:11:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:438:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:438:in `activate!'
/var/www/discourse/lib/discourse.rb:151:in `block in activate_plugins!'
/var/www/discourse/lib/discourse.rb:148:in `each'
/var/www/discourse/lib/discourse.rb:148:in `activate_plugins!'
/var/www/discourse/config/application.rb:207:in `<class:Application>'
/var/www/discourse/config/application.rb:28:in `<module:Discourse>'
/var/www/discourse/config/application.rb:27:in `<top (required)>'
/var/www/discourse/Rakefile:5:in `require'
/var/www/discourse/Rakefile:5:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
I, [2018-04-25T02:02:01.848613 #16]  INFO -- : gem install rufus-scheduler -v 3.1.8 -i /var/www/discourse/plugins/discourse-admin-statistics-digest/gems/2.4.4 --no-document --ignore-dependencies
Successfully installed rufus-scheduler-3.1.8
1 gem installed
gem install sidekiq-scheduler -v 2.0.9 -i /var/www/discourse/plugins/discourse-admin-statistics-digest/gems/2.4.4 --no-document --ignore-dependencies
Successfully installed sidekiq-scheduler-2.0.9
1 gem installed

(Matt Palmer) #2

Looks like the discourse-admin-statistics-digest plugin needs to be updated to depend on at least version 2.1.10 of sidekiq-scheduler (that’s the version when the redis dependency switched from ~> 3 to < 5, >= 3. If you’re able to disable that plugin until it’s updated, that’ll get you back up and running.


(Slack-Moehrle) #3

You are correct. And you noticed this because in the call stack it was the only plugin mentioned?


(Matt Palmer) #4

Yeah, pretty much. Confirmed by looking at the plugin.rb and the dependencies for that version of sidekiq-scheduler. Figuring out which version of sidekiq-scheduler fixed the dependencies was a binary search.


(Sam Saffron) #5

The weird thing is that the plugin even pulls in that dependency when we already ship a battle tested scheduler out of the box


(Matt Palmer) #6

Pre-release code review ftw!


#8

I got the same issue, leaving the discourse unresponsive with following code only:

    Oops

The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.

Detailed information about the error was logged, and an automatic notification generated. We'll take a look at it.

No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site's feedback category.

(Régis Hanol) #9

Have you tried removing the discourse-admin-statistics-digest plugin?


#10

I removed the plugin, the only issue remains the fact that it shows me discourse was upgraded from the upgrade manager, I will need to launch the upgrade again from cli, to fix the issue, or?


(Régis Hanol) #11

Do a ./launcher rebuild app and that’ll fix it :wink:


#12

Restart docker I used this for rebuild, hope it helps.
No this does not helps.


#13

I got this:

root@support:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Updating Launcher
Updating 6b67357..56032fc
error: Your local changes to the following files would be overwritten by merge:
templates/web.ssl.template.yml
Please, commit your changes or stash them before you can merge.
Aborting
failed to update
Ensuring launcher is up to date
Fetching origin
Updating Launcher
Updating 6b67357..56032fc
error: Your local changes to the following files would be overwritten by merge:
templates/web.ssl.template.yml
Please, commit your changes or stash them before you can merge.
Aborting
failed to update
^C

(Régis Hanol) #14

Why have you changed the templates/web.ssl.template.yml file?

You can revert your changes with git checkout -- templates/web.ssl.template.yml


#15

I just enabled the SSL with that file… Seems like checkout helps now its rebuilding the app without the plugin causing the problems.

It seems I have these lines in the rebuild process, no idea why is downloading the plugin again:

I, [2018-04-25T15:41:42.453478 #15] INFO -- : &gt; cd /var/www/discourse/plugins &amp;&amp; git clone https://github.com/discourse/discourse-admin-statistics-digest.git

Cloning into 'discourse-admin-statistics-digest'...

#16
Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES COMPLETE
Old 9.5 database is stored at /shared/postgres_data_old
To complete the upgrade, rebuild again using:
./launcher rebuild app

I got this, was just wondering where are the .sh scripts located?


#17

It seems the plugin is redownloaded every-time I launch the relaunch…

(See full trace by running task with --trace)
I, [2018-04-25T15:51:04.621183 #15]  INFO -- : gem install rufus-scheduler -v 3.1.8 -i /var/www/discourse/plugins/discourse-admin-statistics-digest/gems/2.4.4 --no-document --ignore-dependencies
Successfully installed rufus-scheduler-3.1.8
1 gem installed
gem install sidekiq-scheduler -v 2.0.9 -i /var/www/discourse/plugins/discourse-admin-statistics-digest/gems/2.4.4 --no-document --ignore-dependencies
Successfully installed sidekiq-scheduler-2.0.9
1 gem installed

I, [2018-04-25T15:51:04.621462 #15]  INFO -- : Terminating async processes
I, [2018-04-25T15:51:04.621498 #15]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main pid: 46
I, [2018-04-25T15:51:04.621559 #15]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 162
162:signal-handler (1524671464) Received SIGTERM scheduling shutdown...
2018-04-25 15:51:04.621 UTC [46] LOG:  received fast shutdown request
162:M 25 Apr 15:51:04.698 # User requested shutdown...
162:M 25 Apr 15:51:04.698 * Saving the final RDB snapshot before exiting.
2018-04-25 15:51:04.925 UTC [46] LOG:  aborting any active transactions
162:M 25 Apr 15:51:04.929 * DB saved on disk
162:M 25 Apr 15:51:04.929 # Redis is now ready to exit, bye bye...
2018-04-25 15:51:04.930 UTC [46] LOG:  worker process: logical replication launcher (PID 55) exited with exit code 1
2018-04-25 15:51:04.930 UTC [50] LOG:  shutting down
2018-04-25 15:51:05.078 UTC [46] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 587 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
09d710974b979465a0afad78b9e782305214e201ceed7fcc0c5a19ff3d83f786
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

(Slack-Moehrle) #19

Did you comment it out in /var/discourse/container/app.yml?


Discourse rebuild fails after upgrade to 2.0.0 beta7 (Perl/PostgreSQL issues)
(Wolftune) #21

At one point, right around when this post was first made (but my case is unrelated to the original here, this is my first post on this topic), we ran into a dependency glitch when adding plugins. We got everything back but left off the admin-stats plugin out of precaution (maybe that was at fault for us too).

I decided I’d wait to see if an update resolved things and it was safe to add it back in. Any updates? Not that we have any rush to bother having it…


(Slack-Moehrle) #22

I can say that I still have the same plugin commented out.


(Mick Hellstrom) #23

Just thought I’d add on to this thread…

I went through an upgrade cycle as well and saw the following error in the ./launcher upgrade app output:
FAILED TO BOOTSTRAP

A restart would result in an HTML 502 bad gateway error, with logfile errors saying:
postgres@postgres ERROR: database "discourse" already exists

So I was dead in the water.

One of the solutions: If anyone ever sees an issue with upgrades, the first thing I’d look at is any plugins that have been enabled and see if the GitHub link is accurate. You might see this sort of thing:

I, [2018-06-14T02:31:24.112027 #13]  INFO -- : > cd /var/www/discourse/plugins && git clone https://github.com/discourse/discourse-header-search
Cloning into 'discourse-header-search'...
fatal: could not read Username for 'https://github.com': No such device or address

This just means that someone has changed their GitHub repo name and https://github.com/discourse/discourse-header-search no longer exists, (HTML 404 error).

So, simply remove, (or comment out), any plugins that cause issues and rebuild again.

Bob’s Your Uncle.