Rebuild Failing due to Postgres DB corruption


(Resurrectedstar) #1

Hi, I am getting a rebuild failure as of right now

/pups/lib/pups/exec_command.rb:85:in `spawn': cd /var/www/discourse && sudo -E -u discourse bundle exec rake db:migrate failed with return #<Process::Status: pid 285 exit 1> (RuntimeError)
    from /pups/lib/pups/exec_command.rb:55:in `block in run'
    from /pups/lib/pups/exec_command.rb:53:in `each'
    from /pups/lib/pups/exec_command.rb:53:in `run'
    from /pups/lib/pups/command.rb:6:in `run'
    from /pups/lib/pups/config.rb:85:in `block (2 levels) in run_commands'
    from /pups/lib/pups/config.rb:76:in `each'
    from /pups/lib/pups/config.rb:76:in `block in run_commands'
    from /pups/lib/pups/config.rb:75:in `each'
    from /pups/lib/pups/config.rb:75:in `run_commands'
    from /pups/lib/pups/config.rb:71:in `run'
    from /pups/lib/pups/cli.rb:31:in `run'
    from /pups/bin/pups:8:in `<main>'

Though this might be because I just moved this docker from another host, I am not sure.


(Régis Hanol) #2

We need more information/context to be able to help you out. Can you post the whole log? (remove any password beforehand)


(Resurrectedstar) #3

http://paste.ubuntu.com/7915045/


(Régis Hanol) #4

Hmm, the error seems to be

PG::Error: ERROR:  missing chunk number 0 for toast value 35420 in pg_toast_16571

Seems like your database is somewhat corrupted :frowning:


(Resurrectedstar) #5

Yikes!
And I don’t have any postgres experience to recover or delete the offending row, time to search the internet it seems :stuck_out_tongue:


(Jens Maier) #6

Luckily, this appears to be a common problem… :grin:

Get into Postgres:

cd /var/docker
./launcher enter app
sudo -u postgres psql discourse

Then run this SQL:

REINDEX TABLE pg_toast.pg_toast_16571;
VACUUM ANALYZE posts;
\q

And finally exit.

This should fix the errors you get during migration but the data in your database may have been permanently damaged.


(Kane York) #7

Note that there are multiple scenarios in which corrupted database values will make Discourse unusable, esp. w.r.t. constraints.


(Jens Maier) #8

True, however in this case the damage, if any, should be limited to a raw or cooked column entry.


(system) #10

(Sam Saffron) #11

These days if the DB fails under this mode during upgrade we will provide detailed instructions about what to do. Basically, you need to do a full backup to file and then restore it to the new version.


(Sam Saffron) #12