Latest automatic version upgrade failed


(Franz) #1

Hello,

Today and for the first time ever, the automatic version upgrade failed.

It looks like the bundle exec rake multisite:migrate command failed with the following error:

LoadError: cannot load such file -- memory_profiler

The error is 100% reproducible.

Any idea what’s going on?

Full upgrade log:

$ cd /var/www/discourse && git fetch && git reset --hard HEAD@{upstream}
HEAD is now at 0c6d8e1 Merge pull request #4300 from NuckChorris/patch-2
********************************************************
*** Please be patient, next steps might take a while ***
********************************************************
$ bundle install --deployment --without test --without development
Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
Fetching gem metadata from https://rubygems.org/.............
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Using rake 11.1.2
Using i18n 0.7.0
Using json 1.8.3
Using minitest 5.9.0
Using thread_safe 0.3.5
Using builder 3.2.2
Using erubis 2.7.0
Using mini_portile2 2.1.0
Using pkg-config 1.1.7
Using rack 1.6.4
Using mime-types 2.99.1
Using arel 6.0.3
Using json_pure 1.8.3
Using babel-source 5.8.34
Using execjs 2.7.0
Using ember-source 1.12.2
Installing uniform_notifier 1.9.0
Using byebug 8.2.1
Using certified 1.0.0
Using coderay 1.1.0
Using concurrent-ruby 1.0.2
Using connection_pool 2.2.0
Using crass 1.0.2
Using daemons 1.2.3
Using diff-lcs 1.2.5
Using thor 0.19.1
Installing discourse_fastimage 2.0.3
Using docile 1.1.5
Using unf_ext 0.0.7.1
Using email_reply_trimmer 0.1.3
Using eventmachine 1.2.0.1
Using excon 0.45.4
Using exifr 1.2.4
Using fabrication 2.9.8
Using fakeweb 1.3.0
Using multipart-post 2.0.0
Using fast_blank 1.0.0
Using fast_stack 0.2.0
Using fast_xs 0.8.0
Using ffi 1.9.10
Using flamegraph 0.9.5
Using fspath 2.1.1
Using sorcerer 1.0.2
Using guess_html_encoding 0.0.11
Using hashie 3.4.4
Using highline 1.7.8
Using hiredis 0.6.1
Using htmlentities 4.3.4
Using http_accept_language 2.0.5
Using image_size 1.4.1
Using in_threads 1.3.1
Using progress 3.1.1
Using jwt 1.5.2
Using kgio 2.10.0
Using libv8 5.0.71.48.3
Using listen 0.7.3
Installing logster 1.2.4
Using lru_redux 1.1.0
Using metaclass 0.0.4
Using method_source 0.8.2
Using mock_redis 0.15.4
Using moneta 0.8.0
Using msgpack 0.7.6
Using multi_json 1.11.2
Using multi_xml 0.5.5
Using mustache 1.0.3
Using netrc 0.11.0
Using oauth 0.4.7
Using oj 2.14.3
Using ruby-openid 2.7.0
Using redis 3.3.0
Using pg 0.18.4
Using slop 3.6.0
Using puma 3.2.0
Using r2 0.2.6
Using bundler 1.11.2
Using rails_multisite 1.0.4
Using raindrops 0.16.0
Using rb-fsevent 0.9.7
Using trollop 2.1.2
Using rinku 2.0.0
Using rmmseg-cpp 0.2.9
Installing rspec-support 3.4.1
Using rtlit 0.0.5
Using sass 3.2.19
Using tilt 2.0.2
Using shoulda-context 1.2.1
Using simple-rss 1.3.1
Using simplecov-html 0.10.0
Using spork 1.0.0rc4
Using timecop 0.8.0
Using rake-compiler 0.9.9
Using tzinfo 1.2.2
Installing nokogiri 1.6.8 with native extensions
Using rack-test 0.6.3
Installing message_bus 2.0.1
Using rack-mini-profiler 0.10.1
Using rack-protection 1.5.3
Using mail 2.6.4
Using jmespath 1.2.4
Using babel-transpiler 0.7.0
Using uglifier 3.0.0
Using barber 0.11.1
Using ember-data-source 1.0.0.beta.16.1
Using sprockets 3.6.0
Using unf 0.1.4
Using thin 1.6.4
Using faraday 0.9.2
Using rb-inotify 0.9.7
Using given_core 3.7.1
Using omniauth 1.3.1
Using image_optim 0.20.2
Using mini_racer 0.1.3
Using mocha 1.1.0
Using rack-openid 1.3.1
Using openid-redis-store 0.0.2
Using redis-namespace 1.5.2
Using sidekiq 4.1.2
Using pry 0.10.3
Using unicorn 5.1.0
Using rbtrace 0.4.8
Installing rspec-core 3.4.4
Installing rspec-expectations 3.4.0
Installing rspec-mocks 3.4.1
Using simplecov 0.11.1
Using fast_xor 1.1.3
Using activesupport 4.2.6
Using loofah 2.0.3
Using nokogumbo 1.4.7
Installing onebox 1.5.42
Using ruby-readability 0.7.0
Using sinatra 1.4.6
Using aws-sdk-core 2.3.7
Using ember-handlebars-template 0.7.3
Using domain_name 0.5.25
Using oauth2 1.0.0
Using omniauth-oauth 1.1.0
Using omniauth-openid 1.0.1
Using sidekiq-statistic 1.2.0
Using pry-nav 0.2.4
Using pry-rails 0.3.4
Installing rspec 3.4.0
Using rails-deprecated_sanitizer 1.0.3
Using globalid 0.3.6
Using activemodel 4.2.6
Installing bullet 5.0.0
Using shoulda-matchers 2.8.0
Using rails-html-sanitizer 1.0.3
Using sanitize 4.0.1
Using aws-sdk-resources 2.3.7
Using http-cookie 1.0.2
Using omniauth-oauth2 1.3.1
Using omniauth-twitter 1.2.1
Using rspec-given 3.7.1
Using rspec-html-matchers 0.7.0
Using rails-dom-testing 1.0.7
Using activejob 4.2.6
Using active_model_serializers 0.8.3
Using activerecord 4.2.6
Using rails-observers 0.1.2
Using shoulda 3.5.0
Using aws-sdk 2.3.7
Using rest-client 1.8.0
Using omniauth-facebook 3.0.0
Using omniauth-github-discourse 1.1.2
Using omniauth-google-oauth2 0.3.1
Using omniauth-instagram 1.0.2
Using actionview 4.2.6
Using seed-fu 2.3.5
Using actionpack 4.2.6
Using actionmailer 4.2.6
Using railties 4.2.6
Using sprockets-rails 3.0.4
Using discourse-qunit-rails 0.0.9
Using jquery-rails 4.0.5
Installing rspec-rails 3.4.2
Using rails 4.2.6
Using sass-rails 5.0.4
Using ember-rails 0.18.5
Using spork-rails 4.0.0
Bundle complete! 99 Gemfile dependencies, 180 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into ./vendor/bundle.
$ bundle exec rake multisite:migrate
rake aborted!
LoadError: cannot load such file -- memory_profiler
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/var/www/discourse/config/initializers/006-mini_profiler.rb:7:in `'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `block in load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:652:in `block in load_config_initializer'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:651:in `load_config_initializer'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:616:in `block (2 levels) in '
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `block in '
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `tsort_each_child'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/config/environment.rb:5:in `'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/application.rb:328:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Tasks: TOP => multisite:migrate => environment
(See full trace by running task with --trace)

Franz


(Sam Saffron) #2

grab the latest image … you are probably running a ruby 2.0 image

cd /var/discourse
git pull
./launcher cleanup 
./launcher rebuild app

(Franz) #3

Thanks Sam!

I just upgraded Docker to the latest version (1.11) and I’m in the process of rebuilding the container.

Is it expected that both ./launcher cleanup and ./launcher rebuild app, run in succession, fail to find the Discourse image?

Unable to find image 'discourse/discourse:1.3.5' locally
1.3.5: Pulling from discourse/discourse

I have almost zero experience with Docker…


(Sam Saffron) #4

Are you running the latest version of Docker?


(Franz) #5

As far as I can tell, yes I am:

$ docker --version
Docker version 1.11.2, build b9f10c9

(Sam Saffron) #6

Wait… yeah its pulling the image … this takes a while.


(Franz) #7

Woops, turns out ./launcher rebuild app just failed as well:

Setting up postgresql-9.3 (9.3.13-1.pgdg16.04+1) ...
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Creating new cluster 9.3/main ...
  config /etc/postgresql/9.3/main
  data   /var/lib/postgresql/9.3/main
  locale C
  socket /var/run/postgresql
  port   5433
Warning: The selected stats_temp_directory /var/run/postgresql/9.3-main.pg_stat_tmp
is not writable for the cluster owner. Not adding this setting in
postgresql.conf.
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/9.5/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up postgresql-contrib-9.3 (9.3.13-1.pgdg16.04+1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
 * Stopping PostgreSQL 9.3 database server
   ...done.
 * Stopping PostgreSQL 9.5 database server
 * Error: Config owner (postgres:107) and data owner (root:0) do not match, and config owner is not root
   ...fail!
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for invalid "line" user columns                    ok
Creating dump of global objects                             ok
Creating dump of database schemas
  discourse
  postgres
  template1
                                                            ok

encodings for database "postgres" do not match:  old "SQL_ASCII", new "UTF8"
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

When your instance is running:
Run ./launcher enter app
Run cd /shared/postgres_backup && sudo -u postgres pg_dump discourse > backup.db

Undo the base_image and postgres template from your container config
Run: ./launcher stop app
Run: sudo mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
Run: ./launcher rebuild app

Run: ./launcher enter app
Run: cd /shared/postgres_backup
Run: sv stop unicorn
Run: sudo -iu postgres dropdb discourse
Run: sudo -iu postgres createdb discourse
Run: sudo -iu postgres psql discourse < backup.db
Run: exit
Run: ./launcher rebuild app



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 39 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "/root/upgrade_postgres"
21352e0361bb011208c9e88287294a15f7a49b5b0ce80838815a717d6ebd963f
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Should I follow the suggested instructions?


(Sam Saffron) #8

Yes … you obviously have quite an old version of Discourse there :slight_smile: been running it for a few years I guess.


(Franz) #9

Yes, but my question was, is it expected that it first pulls the image for ./launcher cleanup, and pulls it again for ./launcher rebuild app?


(Franz) #10

Yes I’ve been a happy Discourse users for a few years now.

I’ll follow the suggested instructions then, and hope for the best.

Thanks.


(Franz) #11

Worked flawlessly, Discourse now at v1.6.0.beta10 +9.


(Franz) #12

Tangential question: for some reason (historical, undoubtedly), the root directory for my Discourse instance is /var/docker instead of the more logical /var/discourse.

Is it possible to do the renaming without too much hassle? What would the procedure be like?

EDIT: moved to own topic: https://meta.discourse.org/t/renaming-discourses-root-directory/


(Jakob Borg) #13

I think the script always pulls the latest version, and the cleanup action removes any unused images. Which includes the latest, if you’re not running on it. So yes, it seems expected although somewhat suboptimal.


(Allen - Watchman Monitoring) #14

Indeed - reported here