Upgrade discourse and rebuild app failed


#1

Today, my discourse site alert me to upgrade discourse to latest version.

While I use the website console to upgrade it, It stuck at rake db command

The error log is like this:

$ bundle exec rake multisite:migrate
rake aborted!
LoadError: cannot load such file -- memory_profiler
...

Then whole site down, and ask me to run ./launcher rebuild app.

I enter the machine and run the ./launcher rebuild app command, it raise error and stop. I don’t know where the bug is. May anyone help me to find how to fix it? Thank you.

Error log:

root@localhost:/srv/discourse## ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 10 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2016-07-13T10:54:39.993982 #16]  INFO -- : Loading --stdin
I, [2016-07-13T10:54:40.001144 #16]  INFO -- : > mkdir -p /shared/postgres_run
I, [2016-07-13T10:54:40.004174 #16]  INFO -- :
I, [2016-07-13T10:54:40.004547 #16]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2016-07-13T10:54:40.006645 #16]  INFO -- :
I, [2016-07-13T10:54:40.006988 #16]  INFO -- : > chmod 775 /shared/postgres_run
I, [2016-07-13T10:54:40.009230 #16]  INFO -- :
I, [2016-07-13T10:54:40.009463 #16]  INFO -- : > rm -fr /var/run/postgresql
I, [2016-07-13T10:54:40.011729 #16]  INFO -- :
I, [2016-07-13T10:54:40.012075 #16]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2016-07-13T10:54:40.013559 #16]  INFO -- :
I, [2016-07-13T10:54:40.013935 #16]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2016/07/13 10:54:40 socat[24] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2016-07-13T10:54:40.040462 #16]  INFO -- :
I, [2016-07-13T10:54:40.040768 #16]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2016-07-13T10:54:40.043011 #16]  INFO -- :
I, [2016-07-13T10:54:40.043214 #16]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2016-07-13T10:54:40.045059 #16]  INFO -- :
I, [2016-07-13T10:54:40.045289 #16]  INFO -- : > mkdir -p /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2016-07-13T10:54:40.046984 #16]  INFO -- :
I, [2016-07-13T10:54:40.047232 #16]  INFO -- : > chown postgres:postgres /shared/postgres_run/9.5-main.pg_stat_tmp
I, [2016-07-13T10:54:40.048691 #16]  INFO -- :
I, [2016-07-13T10:54:40.052064 #16]  INFO -- : File > /etc/service/postgres/run  chmod: +x
I, [2016-07-13T10:54:40.055496 #16]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x
I, [2016-07-13T10:54:40.058368 #16]  INFO -- : File > /root/upgrade_postgres  chmod: +x
I, [2016-07-13T10:54:40.058778 #16]  INFO -- : > chown -R root /var/lib/postgresql/9.5/main
I, [2016-07-13T10:54:40.070238 #16]  INFO -- :
I, [2016-07-13T10:54:40.070466 #16]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/9.5/bin/initdb -D /shared/postgres_data || exit 0
I, [2016-07-13T10:54:40.072063 #16]  INFO -- :
I, [2016-07-13T10:54:40.072238 #16]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2016-07-13T10:54:40.081231 #16]  INFO -- :
I, [2016-07-13T10:54:40.081469 #16]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2016-07-13T10:54:40.083281 #16]  INFO -- :
I, [2016-07-13T10:54:40.083616 #16]  INFO -- : > /root/upgrade_postgres
initdb: invalid locale settings; check LANG and LC_* environment variables
I, [2016-07-13T10:54:40.184797 #16]  INFO -- : Upgrading PostgreSQL to from version 9.3 to 9.5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.


I, [2016-07-13T10:54:40.185226 #16]  INFO -- : > rm /root/upgrade_postgres
I, [2016-07-13T10:54:40.187091 #16]  INFO -- :
I, [2016-07-13T10:54:40.187597 #16]  INFO -- : Replacing data_directory = '/var/lib/postgresql/9.5/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.188630 #16]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.192082 #16]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.192830 #16]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.193459 #16]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.194030 #16]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.195198 #16]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2016-07-13T10:54:40.198041 #16]  INFO -- :
I, [2016-07-13T10:54:40.198657 #16]  INFO -- : Replacing (?-mix:#?max_wal_senders *=.*) with max_wal_senders = $db_max_wal_senders in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.199451 #16]  INFO -- : Replacing (?-mix:#?wal_level *=.*) with wal_level = $db_wal_level in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.199927 #16]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.200505 #16]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.201019 #16]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/9.5/main/postgresql.conf
I, [2016-07-13T10:54:40.201624 #16]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/9.5/main/pg_hba.conf
I, [2016-07-13T10:54:40.202403 #16]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/9.5/main/pg_hba.conf
I, [2016-07-13T10:54:40.202885 #16]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main
I, [2016-07-13T10:54:40.208821 #16]  INFO -- : > sleep 5
LOG:  skipping missing configuration file "/shared/postgres_data/postgresql.auto.conf"
2016-07-13 10:54:40 UTC [51-2] FATAL:  database files are incompatible with server
2016-07-13 10:54:40 UTC [51-3] DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.5.3.
I, [2016-07-13T10:54:45.211363 #16]  INFO -- :
I, [2016-07-13T10:54:45.212043 #16]  INFO -- : > su postgres -c 'createdb discourse' || true
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "zh_TW.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
createdb: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2016-07-13T10:54:45.297910 #16]  INFO -- :
I, [2016-07-13T10:54:45.298363 #16]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "zh_TW.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2016-07-13T10:54:45.380898 #16]  INFO -- :
I, [2016-07-13T10:54:45.381258 #16]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "zh_TW.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2016-07-13T10:54:45.466160 #16]  INFO -- :
I, [2016-07-13T10:54:45.466429 #16]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "zh_TW.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2016-07-13T10:54:45.549852 #16]  INFO -- :
I, [2016-07-13T10:54:45.550500 #16]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 77 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
7c2805d7f223b77b5d0ac4815a043b2c5f8a0eece349c91a468699c56de2cd36
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

#2

Might it be linked to your locale settings ? Do a nano containers/app.yml change the language to english for now.

Then :

git pull
./launcher rebuild app

#3

Thank you.

After I change the language setting to en_US.UTF-8, it shows instruction:

-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

You are going to need to export your data and import into a clean instance:

In containers/app.yml: Change "templates/postgres.template.yml" TO "templates/postgres.9.3.template.yml"

Run ./launcher rebuild app again

After that, the server back.

Thank you!!