Issue Rebuilding App Failing on Postgres Upgrade

AI-generated summary

Users are encountering failures during Discourse app rebuilds due to PostgreSQL upgrades, specifically when upgrading from version 9.5 to 10. The error indicates insufficient disk space, despite users reporting ample free space (e.g., 23GB or 26GB available). Jose_C_Gomez initially faced the issue and resolved it temporarily by reverting to the older PostgreSQL template, though the root cause remained unclear. pfaffman suggested checking disk usage inside the container, which Jose_C_Gomez confirmed showed ~20GB free. itsbhanusharma and codinghorror clarified that PostgreSQL requires roughly 2x the uncompressed database size for upgrades — not total disk space — which may explain the failure even with apparent free space. Jose_C_Gomez noted backups are ~3GB compressed, implying ~10GB uncompressed. pfaffman recommended using du -h /var/discourse/shared/standalone/postgres_data to check actual DB size. Ignacio_Carrera reported a similar issue with a 329MB database and 26GB free space, suggesting the problem may persist even with seemingly sufficient space. Others, including dhoop and pzubkiewicz, confirmed encountering the same error. pfaffman referenced a PostgreSQL 12 update guide for additional context. The core issue appears to be PostgreSQL’s space requirement during upgrades, which may not align with users’ perception of available space.

Trying to do a launcher rebuild and getting this error

I, [2018-06-30T18:56:08.346656 #17]  INFO -- :
I, [2018-06-30T18:56:08.346728 #17]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2018-06-30T18:56:08.357457 #17]  INFO -- :
I, [2018-06-30T18:56:08.357700 #17]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2018-06-30T18:56:08.359892 #17]  INFO -- :
I, [2018-06-30T18:56:08.360109 #17]  INFO -- : > /root/upgrade_postgres
I, [2018-06-30T18:56:08.374537 #17]  INFO -- : Upgrading PostgreSQL from version 9.5 to 10
WARNING: Upgrading PostgresSQL would require an addtional 13M of disk space
Please free up some space, or expand your disk, before continuing.

To avoid upgrading change "templates/postgres.template.yml" TO "templates/postgres.9.5.template.yml" in containers/app.yml



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 68 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "/root/upgrade_postgres"
a6266e7beac5a72d92ef181ff8c75b6999b94f5c7831622943c7773a04652c5d
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

It appears to be failing upgrading PostGre because it needs more space? I have plenty of space 23GB

root@EPICORHELP:/var/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           396M  364K  395M   1% /run
/dev/vda1        79G   53G   23G  71% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user

Any ideas?

Renaming the template as suggested in the error worked… However I don’t understand why its failing. I have plenty of space.

Enter the container and do

df - h

Ok… here’s the output… Looks fine?

root@EPICORHELP:/var/discourse# ./launcher enter app
root@EPICORHELP-app:/var/www/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
none             79G   55G   20G  74% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        79G   55G   20G  74% /shared
shm             512M  4.0K  512M   1% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/firmware

You’d need 2x your used space as free space for postgres upgrade to work …

I see 55GB used and 79GB available.
Maybe try upgrade with 170GB storage or more?

2x the database size specifically, to be clear.

Any way to find out the exact database size?

When I take backups within discourse the file generated is 3GB…

With compression, so at least 10gb uncompressed.

du - h /var/discourse/shared/standalone/postgres_data

(from memory and on my phone, so, no promises)

Same problem here:

Run: ./launcher rebuild app

FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 66 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "/root/upgrade_postgres"
ba104fb4aff36d211ccb42e81365851f64e38219f0d00a336051643fd4255740
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

And my DB is ~330mb

# du -h /var/discourse/shared/standalone/postgres_data
…
329M    /var/discourse/shared/standalone/postgres_data

Free space is 26GiB

# df -H
…
/dev/vda         50G   22G   26G  46% /

@Ignacio_Carrera Did you ever figure this out? I have the same problem

I have a same problem as well :frowning:

Did you see PostgreSQL 12 update?