Bootstrap failed, please help :(

[amotl (Andreas Motl)]

Hi there,

@wishspinner

LOG:  database system shutdown was interrupted; last known up at 2020-04-11 12:42:48 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  invalid record length at ...

we just received the same message on a rebuild for a version upgrade …

@pfaffman

Looks like your database is corrupt.

… and thought the same was happening to us – for the first time ever –

@pfaffman

The easiest solution is probably to delete the database directory, which will build a new database. Then you can restore from your most recent backup.

… and hesitantly looked for the backup directory ;]. A fresh backup was there, so we relaxed a bit.

@wishspinner

To give you an update now that I’m out of the heat of it - rebooting the server allowed the app to rebuild.

We also just wanted to share that, while we did not reboot the server, we waited a bit, failed rebuilding a second time with the same error message invalid record length at, but now the update process seems to be working when trying a third time. The server has enough disk space, but is currently a bit under pressure I/O-wise.

Maybe this helps others who may be running into the same situation. From this experience, we would advise to a) mitigate resource overconsumption on the host system before trying another rebuild, and b) not to give up too early.

With kind regards,
Andreas.