Restoring from backup


(Thomvil) #1

Yesterday my discourse stopped working, after a user tried to post a message and failed.

I tried the usual

cd /var/discourse
git pull
./launcher rebuild app

It complained that i needed a newer Docker. So i upgraded with

wget -qO- https://get.docker.com/ | sh

My docker info is now:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 7
Server Version: 17.10.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 12
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-79-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.953GiB
Name: 
ID: JU2V:SOZZ:UZ3D:U7HB:SGQO:EX5V:HEHT:7CSO:A6YN:ILM6:L7QQ:XS7S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Rebuilding now fails with a ScriptTerminatedError:


2017-11-17 20:12:11.625 UTC [916] discourse@discourse LOG:  duration: 139.534 ms  statement:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
	                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
	                     c.collname, col_description(a.attrelid, a.attnum) AS comment
	                FROM pg_attribute a
	                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
	                LEFT JOIN pg_type t ON a.atttypid = t.oid
	                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
	               WHERE a.attrelid = '"users"'::regclass
	                 AND a.attnum > 0 AND NOT a.attisdropped
	               ORDER BY a.attnum
	
I, [2017-11-17T20:13:27.161531 #883]  INFO -- : Writing /var/www/discourse/public/assets/vendor-057c8aaf728aa3bc264d5ff93a70e003efe4f4ecdc16ba0a02d452e6e3ef4402.js
rake aborted!
MiniRacer::ScriptTerminatedError: JavaScript was terminated (either by timeout or explicitly)
...
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2017-11-17T20:13:59.947263 #13]  INFO -- : Purging temp files
Bundling assets

I, [2017-11-17T20:14:00.011489 #13]  INFO -- : Terminating async processes
I, [2017-11-17T20:14:00.018790 #13]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main pid: 42
I, [2017-11-17T20:14:00.023107 #13]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 156
2017-11-17 20:14:00.037 UTC [42] LOG:  received fast shutdown request
156:signal-handler (1510949640) Received SIGTERM scheduling shutdown...
156:M 17 Nov 20:14:00.056 # User requested shutdown...
156:M 17 Nov 20:14:00.062 * Saving the final RDB snapshot before exiting.
2017-11-17 20:14:00.074 UTC [42] LOG:  aborting any active transactions
2017-11-17 20:14:00.076 UTC [49] LOG:  autovacuum launcher shutting down
2017-11-17 20:14:00.274 UTC [46] LOG:  shutting down
2017-11-17 20:14:00.506 UTC [46] LOG:  database system is shut down
I, [2017-11-17T20:14:10.057693 #13]  INFO -- : exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid:156 did not terminate cleanly, forcing termination!

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 881 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
9ed892bf5e6fb2308d24e32c0550216abed9c79f887a5dab2fea08b65e3eb5bb
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Entire log available as gist

From that error, i assume there is a problem with the post that caused the initial failure.

(Or did i mess up something with docker?)

I am not able to enter with ./launcher enter app. Is there a way to restore from the latest backup i find in /var/discourse/shared/standalone/backups?

Thank you for any help.


(Paul) #2

@thomvil you could take a look at this thread Move your Discourse Instance to a Different Server as it gives the syntax for command line restores. Never used it so I can’t say it will work.