Hello everyone, I’m running into an issue with PostgreSQL startup issue when I tried to rebuild my app, and I’m hoping to get some help. Here’s the log, it has been stuck on this status for more than 30 mins. Status: Image is up to date for discourse/base:2.0.20240825-0027 docker.io/discourse/base:2.0.20240825-0027 /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups.rb /usr/local/bin/pups --stdin I, [2024-08-26T17:16:15.344712 #1] INFO -- : Reading from stdin I, [2024-08-26T17:16:15.357924 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2024-08-26T17:16:15.362740 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2024-08-26T17:16:15.367767 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2024-08-26T17:16:15.372845 #1] INFO -- : File > /root/install_postgres chmod: +x chown: I, [2024-08-26T17:16:15.377501 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2024-08-26T17:16:15.377876 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.378854 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.379386 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.379835 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.380263 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.380761 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.381203 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.381901 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.382352 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2024-08-26T17:16:15.382802 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2024-08-26T17:16:15.383231 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2024-08-26T17:16:15.383604 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2024-08-26T17:16:15.384079 #1] INFO -- : > if [ -f /root/install_postgres ]; then /root/install_postgres && rm -f /root/install_postgres elif [ -e /shared/postgres_run/.s.PGSQL.5432 ]; then socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 fi 2024/08/26 17:16:15 socat[28] E connect(, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused I, [2024-08-26T17:16:15.452500 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2024-08-26T17:16:15.453058 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2024-08-26T17:16:15.455944 #1] INFO -- : Terminating async processes 2024-08-26 17:16:15.500 UTC [30] LOG: starting PostgreSQL 13.16 (Debian 13.16-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit 2024-08-26 17:16:15.501 UTC [30] LOG: listening on IPv4 address "0.0.0.0", port 5432 2024-08-26 17:16:15.501 UTC [30] LOG: listening on IPv6 address "::", port 5432 2024-08-26 17:16:15.507 UTC [30] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2024-08-26 17:16:15.516 UTC [31] LOG: database system was interrupted; last known up at 2024-08-26 17:10:28 UTC 2024-08-26 17:16:15.769 UTC [31] LOG: database system was not properly shut down; automatic recovery in progress 2024-08-26 17:16:15.774 UTC [31] LOG: redo starts at 18F/E62D1458 2024-08-26 17:16:15.774 UTC [31] LOG: invalid record length at 18F/E62D1490: wanted 24, got 0 2024-08-26 17:16:15.774 UTC [31] LOG: redo done at 18F/E62D1458 2024-08-26 17:16:15.809 UTC [30] LOG: database system is ready to accept connections```

@here It looks like older versions of Docker is incompatible with the newer Docker image which we have released. Here are the steps to fix the problem: apt update apt install docker.io cd /var/discourse ./launcher rebuild app I’ve also bumped our launcher script to abort if we do not at least detect Docker 20.10.0 running.

[imagen] kukulaka: d VM has 125 GiB allocated Darn. It was going to explain everything. :person_shrugging:

Could it be a db size issue? The database on our prod server is pretty large, but dev is very small. That’s the only real difference between the VMs that have successfully upgraded and the affected one (in my case).

Maybe, have you changed the memory config for the database? How big is the database?

I’ll check it out and see if it’s been changed

[imagen] Alan Tan: Btw for anyone running into this problem, you can work around it for now by adding base_image: discourse/base:2.0.20240708-0023 to the top of containers/app.yml file. This is the only thing that worked for me. Thank you for sharing this!! My customers thank you, too :slight_smile: Hoping we can get a proper fix for this soon.

[imagen] Jay Pfaffman: That sounds like a RAM issue. How much ram and swap do you have? I would add a GB or two of SWAP space (and maybe add ram if you have only 1GB) Hello, I just upsized Droplet by doubling RAM and increasing disk size. I’m still facing the same issue. Anything else to try? # free -h total used free shared buff/cache available Mem: 1.9Gi 289Mi 83Mi 11Mi 1.6Gi 1.5Gi Swap: 2.0Gi 3.0Mi 2.0Gi # df -h Filesystem Size Used Avail Use% Mounted on udev 941M 0 941M 0% /dev tmpfs 198M 1.1M 197M 1% /run /dev/vda1 34G 14G 21G 39% / tmpfs 986M 0 986M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 986M 0 986M 0% /sys/fs/cgroup /dev/vda15 105M 7.4M 97M 8% /boot/efi /dev/loop1 56M 56M 0 100% /snap/core18/2829 /dev/loop2 56M 56M 0 100% /snap/core18/2823 /dev/loop3 92M 92M 0 100% /snap/lxd/29619 /dev/loop0 64M 64M 0 100% /snap/core20/2264 /dev/loop4 64M 64M 0 100% /snap/core20/2318 /dev/loop5 39M 39M 0 100% /snap/snapd/21465 /dev/loop6 92M 92M 0 100% /snap/lxd/24061 /dev/loop7 39M 39M 0 100% /snap/snapd/21759 tmpfs 198M 0 198M 0% /run/user/0 overlay 34G 14G 21G 39% /var/lib/docker/overlay2/3c7ebf42647de2b5df34cba2b047079fd3454ea7fe9b04c7b70f227df1e7eafe/merged

[imagen] Alan Tan: Btw for anyone running into this problem, you can work around it for now by adding base_image: discourse/base:2.0.20240708-0023 to the top of containers/app.yml file. OMG! Why didn’t I read this solution before. It also worked for me. So what is the solution going forward? Do we need to keep specifying this base image in the future as well or change it to get updated image?

This is a workaround for now until we have properly identified what is the cause of the problem for a selected group of users like yourself. The difficulty for us now is that not all users are experiencing this problem and we do not have a reliable reproduction of the issue.

Does the problem go away if you move onto the latest beta release?

Surely that’s what they are trying to do by upgrading? :sweat_smile: (NB there is no beta branch anymore )

PostgreSQL atascado durante la reconstrucción

Soporte del producto Instalación

Ed_S (Ed S) 4 Septiembre, 2024 08:48 56

oh, interesante, gracias

Tema		Respuestas	Vistas
Forum Full Crash (Test Pressing) Self-hosting	8	572	4 Septiembre 2024
Stuck on update Self-hosting	2	359	4 Septiembre 2024
Update Error Self-hosting	6	349	30 Agosto 2024
Issue with rebuilding Self-hosting	2	118	15 Septiembre 2026
Postgres Errors on Rebuild Self-hosting	2	1769	16 Septiembre 2015

PostgreSQL atascado durante la reconstrucción

Temas relacionados