Multisite upgrade failed due to missing database columns

bug
1

Doing a ./launcher rebuild discourse-web.yml I got this error:

Compiling css for kulturladen
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column "uploaded_background_id" does not exist
LINE 1: ...categories"."updated_at" FROM "categories" WHERE (uploaded_b...
                                                             ^
: SELECT  "categories"."updated_at" FROM "categories" WHERE (uploaded_background_id IS NOT NULL)  ORDER BY updated_at desc LIMIT 1
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `exec'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/pg.rb:90:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/var/www/discourse/lib/freedom_patches/fast_pluck.rb:41:in `select_raw'
/var/www/discourse/lib/freedom_patches/fast_pluck.rb:79:in `pluck'
/var/www/discourse/app/models/category.rb:108:in `last_updated_at'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/delegation.rb:70:in `block in last_updated_at'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:302:in `scoping'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/delegation.rb:70:in `last_updated_at'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:163:in `digest'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:153:in `stylesheet_filename'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:129:in `stylesheet_fullpath'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:80:in `compile'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:39:in `block in compile'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:36:in `synchronize'
/var/www/discourse/lib/sass/discourse_stylesheets.rb:36:in `compile'
/var/www/discourse/lib/tasks/assets.rake:51:in `block (3 levels) in <top (required)>'
/var/www/discourse/lib/tasks/assets.rake:50:in `each'
/var/www/discourse/lib/tasks/assets.rake:50:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:126:in `block in each_connection'


/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:124:in `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:124:in `each_connection'
/var/www/discourse/lib/tasks/assets.rake:44:in `block in <top (required)>'
/var/www/discourse/lib/tasks/assets.rake:130:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'
PG::UndefinedColumn: ERROR:  column "uploaded_background_id" does not exist
LINE 1: ...categories"."updated_at" FROM "categories" WHERE (uploaded_b...

I’m at the version: tests-passed and using a multi site installation with separate web and data containers.

Looks like the database column `` got added 2 days ago:

https://github.com/discourse/discourse/blob/53086fdb988d157cacbda1f52935f7ceda14812e/db/migrate/20161202034856_add_uploads_to_categories.rb

Is there any way to upgrade the database scheme manually so I get my sites up again?

2

Did you run the migrations on your data container before rebuilding the web container?

3

No because I thought that is taken care of in the web containers YAML file by this hook:

  after_bundle_exec:
    - exec: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate

If not: how am I supposed to migrate the data container?

4

Does rebuilding again fail as well?

Otherwise, we’re need to check if the migrations ran by running the following query in Postgres

SELECT * FROM schema_migration_details WHERE name = 'AddUploadsToCategories';

5

Rebuilding again fails as well. Did it 4 times.

The SELECT statement yields 0 records:

discourse_kulturladen=# SELECT * FROM schema_migration_details WHERE name = 'AddUploadsToCategories';
 id | version | name | hostname | git_version | rails_version | duration | direction | created_at
----+---------+------+----------+-------------+---------------+----------+-----------+------------
(0 rows)
6

BTW: At the default multi site the migration did run:

discourse=# SELECT * FROM schema_migration_details WHERE name = 'AddUploadsToCategories';
 id  |    version     |          name          |   hostname   |               git_version                | rails_version | duration | direction |         created_at
-----+----------------+------------------------+--------------+------------------------------------------+---------------+----------+-----------+----------------------------
 695 | 20161202034856 | AddUploadsToCategories | d1bcdd8eb4d7+| 20f9aad539bc4929f9149ddf989fc8abb69329d3 | 4.2.7.1       |      129 | up        | 2016-12-14 14:10:12.342404
     |                |                        |              |                                          |               |          |           |
(1 row)
7

Try running multisite:migrate manually.

8

I tried from within the data container. It’s code stays at 9th of Dec though (02b21a26ddfe48536251276c235c6c9ac9a7227a).

I probably need to run it from the web container which I can not access because the rebuild failed.

9

Hmm ok I think I see where the problem is.

https://github.com/discourse/discourse_docker/blob/f92ac1191a1a86526a6e92c7ddbad62d44550212/templates/web.template.yml#L178-L184

You’re only running multisite:migrate after the bundle exec hook but during the hook, assets are already being precompiled when the multisite DB hasn’t been migrated yet. To work around the problem, can you change your template to something like

  before_bundle_exec:
    - exec: su discourse -c 'bundle install --deployment --verbose --without test --without development'
    - exec: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate
2 Likes
10

You are right - precompiling assets started before database migration. Adding your code above to the before_bundle_exec hook worked. Thanks a lot for your help!

I had to modify it slightly to get it executed properly. My hook for multi site looks like this now:

  before_bundle_exec:
    - file:
        path: $home/config/multisite.yml
        contents: |
         # (your multisite.yml config here)
    - exec: cd /var/www/discourse && sudo -E -u discourse bundle install --deployment --verbose --without test --without development
    - exec: cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate

Should I consider this as a one time workaround or can this two exec stay forever in my template file?

5 Likes
11

I’m glad I found this topic. Had the same problem. But even now I have an error:

INFO -- : Running `bundle install --deployment --verbose --without "development"` with bundler 1.13.7
The --deployment flag requires a Gemfile.lock. Please make sure you have checked
your Gemfile.lock into version control before deploying.

So it seems it’s missing that Gemfile.lock file.

In the end I fixed it by running the older version of Discourse I had, then:

  • enter it with launcher enter app
  • cd db/migrations
  • paste the contents of the migration file above into a new file
  • cd /var/www/discourse && sudo -E -u discourse bundle exec rake multisite:migrate

Now you can upgrade as usual with launcher rebuild app.

4 Likes