Admin/docker url went away on my site


(Tobias Eigen) #1

Last week sometime, I think on Friday, I did an update to discourse and since then the admin/docker url has gone away. I am unable to do updates via the web ui anymore. Anyone have any ideas about how that might have happened?


(Michael Downey) #2

Try /admin/upgrade.


(Tobias Eigen) #3

also doesn’t work… I get “The page you requested doesn’t exist or is private.”


(Jeff Atwood) #4

This is the second report we’ve had of this – it implies you don’t have admin status on your account, or are not logged in as admin?


(Tobias Eigen) #5

I already thought of that too - I do have admin status and am logged in as admin. I can see the admin settings. One thing that I am now realizing I am also not able to see anymore is the “impersonate user” button - where is that supposed to be?


(Jeff Atwood) #6

@sam did this somehow get tied to Developer level flags on an account?


(Sam Saffron) #7

I just got latest on samsaffron.com … ensured I was not a developer and had access to the page.

@tobiaseigen can you try

  1. To do a full rebuild (cd /var/discourse && git pull && ./launcher rebuild app)
  2. If that fails shoot off the “passwords removed” container config you have, you may have something incompatible there

(Tobias Eigen) #8

Hi @sam thanks for looking into this. I just ran what you suggested and am now getting some gnarly errors that I don’t know what to do with - see below with passwords removed. Ends with FAILED TO BOOTSTRAP. Site is indeed down.

I’d be grateful for any pointers.

root@discourse:/var/discourse# ./launcher rebuild app
WARNING: No swap limit support
Updating discourse docker
Already up-to-date.
Stopping old container
844f86b06bb48626dbbf4bcf9de390c32a7116b9660835107e9e12b8c7cf32b9
Calculated ENV: -e LANG=en_US.UTF-8 -e HOME=/root -e RAILS_ENV=production -e UNICORN_WORKERS=2 -e UNICORN_SIDEKIQS=1 -e RUBY_GC_MALLOC_LIMIT=40000000 -e RUBY_HEAP_MIN_SLOTS=800000 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e DISCOURSE_DEVELOPER_EMAILS=xx -e DISCOURSE_HOSTNAME=forum.kabissa.org -e DISCOURSE_SMTP_ADDRESS=smtp.mandrillapp.com -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=xx -e DISCOURSE_SMTP_PASSWORD=xx
cd /pups && git pull && /pups/bin/pups --stdin
Already up-to-date.
I, [2014-10-07T00:21:08.607027 #42]  INFO -- : Loading --stdin
I, [2014-10-07T00:21:08.619788 #42]  INFO -- : > echo cron is now included in base image, remove from templates
I, [2014-10-07T00:21:08.624596 #42]  INFO -- : cron is now included in base image, remove from templates

I, [2014-10-07T00:21:08.626158 #42]  INFO -- : > mkdir -p /shared/postgres_run
I, [2014-10-07T00:21:08.630461 #42]  INFO -- : 
I, [2014-10-07T00:21:08.632186 #42]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2014-10-07T00:21:08.637148 #42]  INFO -- : 
I, [2014-10-07T00:21:08.638650 #42]  INFO -- : > chmod 775 /shared/postgres_run
I, [2014-10-07T00:21:08.642478 #42]  INFO -- : 
I, [2014-10-07T00:21:08.644045 #42]  INFO -- : > rm -fr /var/run/postgresql
I, [2014-10-07T00:21:08.648186 #42]  INFO -- : 
I, [2014-10-07T00:21:08.649619 #42]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2014-10-07T00:21:08.653474 #42]  INFO -- : 
I, [2014-10-07T00:21:08.655001 #42]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2014/10/07 00:21:08 socat[51] E connect(4, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2014-10-07T00:21:08.663108 #42]  INFO -- : 
I, [2014-10-07T00:21:08.664712 #42]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2014-10-07T00:21:08.670123 #42]  INFO -- : 
I, [2014-10-07T00:21:08.671581 #42]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2014-10-07T00:21:08.676836 #42]  INFO -- : 
I, [2014-10-07T00:21:08.686633 #42]  INFO -- : File > /etc/service/postgres/run  chmod: +x
I, [2014-10-07T00:21:08.696127 #42]  INFO -- : File > /root/upgrade_postgres  chmod: +x
I, [2014-10-07T00:21:08.698734 #42]  INFO -- : > chown -R root /var/lib/postgresql/9.3/main
I, [2014-10-07T00:21:08.900228 #42]  INFO -- : 
I, [2014-10-07T00:21:08.901562 #42]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/9.3/bin/initdb -D /shared/postgres_data || exit 0
I, [2014-10-07T00:21:08.907592 #42]  INFO -- : 
I, [2014-10-07T00:21:08.908479 #42]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2014-10-07T00:21:08.931667 #42]  INFO -- : 
I, [2014-10-07T00:21:08.933035 #42]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2014-10-07T00:21:08.939326 #42]  INFO -- : 
I, [2014-10-07T00:21:08.941155 #42]  INFO -- : > /root/upgrade_postgres
I, [2014-10-07T00:21:08.951556 #42]  INFO -- : 
I, [2014-10-07T00:21:08.953042 #42]  INFO -- : > rm /root/upgrade_postgres
I, [2014-10-07T00:21:08.957304 #42]  INFO -- : 
I, [2014-10-07T00:21:08.958516 #42]  INFO -- : Replacing data_directory = '/var/lib/postgresql/9.3/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.961155 #42]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.962906 #42]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.964702 #42]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.966426 #42]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.968312 #42]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.970032 #42]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.972277 #42]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2014-10-07T00:21:08.978642 #42]  INFO -- : 
I, [2014-10-07T00:21:08.988290 #42]  INFO -- : Replacing (?-mix:#?max_wal_senders *=.*) with max_wal_senders = 4 in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.990116 #42]  INFO -- : Replacing (?-mix:#?wal_level *=.*) with wal_level = hot_standby in /etc/postgresql/9.3/main/postgresql.conf
I, [2014-10-07T00:21:08.992035 #42]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/9.3/main/pg_hba.conf
I, [2014-10-07T00:21:08.993815 #42]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/9.3/main/pg_hba.conf
I, [2014-10-07T00:21:08.996088 #42]  INFO -- : > su - postgres -c '/usr/lib/postgresql/9.3/bin/postmaster -D /etc/postgresql/9.3/main'
I, [2014-10-07T00:21:08.999765 #42]  INFO -- : > sleep 5
2014-10-07 00:21:09 UTC LOG:  database system was interrupted; last known up at 2014-10-07 00:10:36 UTC
2014-10-07 00:21:09 UTC LOG:  invalid primary checkpoint record
2014-10-07 00:21:09 UTC LOG:  invalid secondary checkpoint record
2014-10-07 00:21:09 UTC PANIC:  could not locate a valid checkpoint record
2014-10-07 00:21:09 UTC LOG:  startup process (PID 73) was terminated by signal 6: Aborted
2014-10-07 00:21:09 UTC LOG:  aborting startup due to startup process failure
I, [2014-10-07T00:21:14.009719 #42]  INFO -- : 
I, [2014-10-07T00:21:14.012247 #42]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2014-10-07T00:21:14.134398 #42]  INFO -- : 
I, [2014-10-07T00:21:14.136295 #42]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2014-10-07T00:21:14.279535 #42]  INFO -- : 
I, [2014-10-07T00:21:14.281211 #42]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2014-10-07T00:21:14.424231 #42]  INFO -- : 
I, [2014-10-07T00:21:14.425961 #42]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2014-10-07T00:21:14.565458 #42]  INFO -- : 


FAILED
--------------------
RuntimeError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 97 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:85:in `spawn'
exec failed with the params "su postgres -c 'psql discourse -c \"alter schema public owner to discourse;\"'"
9e2afbc53727c0d8327c39cbb0538a79203066366170a6a1ff182393c0956541
FAILED TO BOOTSTRAP

EDIT: looks like my issue is similar to this - perhaps I had two containers running simultaneously? Don’t know how I might have caused that to happen or how to avoid it in the future. Should I just restore from recent backup?


(Sam Saffron) #9

This is not good … you have corrupted DB Can't bootstrap: "PANIC: could not locate a valid checkpoint record" … you need to run pg_resetxlog on it.

Something along the lines of (let me know the exact command you run once working)

docker run -it --rm samsaffron/discourse -v /var/discourse/shared:/shared /bin/bash
su - postgres
path/to/pg_resetxlog /shared/path/to/db [-f] 

(Tobias Eigen) #10

hmm… this is all a bit beyond me, unfortunately. Thanks for offering these suggestions but I need more specific instructions, or I need to just recover from a backup.

I’m on a digitalocean docker install, all clean without any modifications or plugins. The last time I ran updates was on Friday and I’m up to date on tests-passed as of Friday.


(Sam Saffron) #11

A backup restore is probably a bit easier here.


(Tobias Eigen) #12

After restoring from backup and updating to tests-passed, this problem went away.


(Kane York) #13

Could we get a launcher command to do that - launch a shell without the container running?

Something like ./launcher emergshell app or something.


(Sam Saffron) #14

issue is that it would fail to run DB so only useful very specific diagnostics.

that said ./laucher repairdb app would be SUPER useful if you feel like working it :slight_smile:


(cosy) #15

Have same “E connect(4, AF=1 “/shared/postgres_run/.s.PGSQL.5432”, 36): No such file or directory” on normal installation https://meta.discourse.org/t/failed-to-install-discourse-in-ubuntu14-04-docker-problems/