Eaerlier this morning I tried to stop and rebuild the app but received several errors. The stop didn’t report any errors. The rebuild came with the following errors: 2023-09-04 08:30:19.937 UTC [45] LOG: database system was interrupted; last known up at 2023-09-03 21:07:49 UTC 2023-09-04 08:30:20.045 UTC [45] LOG: invalid resource manager ID in primary checkpoint record 2023-09-04 08:30:20.045 UTC [45] PANIC: could not locate a valid checkpoint record 2023-09-04 08:30:20.859 UTC [42] LOG: startup process (PID 45) was terminated by signal 6: Aborted 2023-09-04 08:30:20.859 UTC [42] LOG: aborting startup due to startup process failure 2023-09-04 08:30:20.919 UTC [42] LOG: database system is shut down Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" bootstrap failed with exit code 2 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. Based on the logs it looks like it has something to do with this error? PANIC: could not locate a valid checkpoint record However, I’m not sure if this is really the case. Let alone what to do about it. Anyone suggestions on what could be wrong? And ideally also a way to get this working (again)? Thnak you,

[image] Will: database system was interrupted; last known up at 2023-09-03 21:07:49 UTC invalid resource manager ID in primary checkpoint record could not locate a valid checkpoint record database system is shut down I’m afraid your database is corrupted. I’d try recovering from a recent backup.

Could you scroll up and check for errors that could provide more detail? (or paste the output here so we can see)

Below the output as a whole. Is this what you where looking for? root@forum:/var/discourse# ./launcher rebuild app x86_64 arch detected. Ensuring launcher is up to date Fetching origin Launcher is up-to-date Stopping old container + /usr/bin/docker stop -t 600 app app 2.0.20230825-1850: Pulling from discourse/base Digest: sha256:3fd7032bbeb6f322ad7138ed154c0fc6177d75ebffba44871abea86620c0c7d5 Status: Image is up to date for discourse/base:2.0.20230825-1850 docker.io/discourse/base:2.0.20230825-1850 /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups.rb /usr/local/bin/pups --stdin I, [2023-09-04T10:52:25.973226 #1] INFO -- : Reading from stdin I, [2023-09-04T10:52:25.978378 #1] INFO -- : > locale-gen $LANG && update-locale I, [2023-09-04T10:52:26.010346 #1] INFO -- : Generating locales (this might take a while)... Generation complete. I, [2023-09-04T10:52:26.010590 #1] INFO -- : > mkdir -p /shared/postgres_run I, [2023-09-04T10:52:26.014888 #1] INFO -- : I, [2023-09-04T10:52:26.015219 #1] INFO -- : > chown postgres:postgres /shared/postgres_run I, [2023-09-04T10:52:26.018839 #1] INFO -- : I, [2023-09-04T10:52:26.019232 #1] INFO -- : > chmod 775 /shared/postgres_run I, [2023-09-04T10:52:26.022489 #1] INFO -- : I, [2023-09-04T10:52:26.022851 #1] INFO -- : > rm -fr /var/run/postgresql I, [2023-09-04T10:52:26.025811 #1] INFO -- : I, [2023-09-04T10:52:26.026082 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql I, [2023-09-04T10:52:26.029384 #1] INFO -- : I, [2023-09-04T10:52:26.029659 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1 2023/09/04 10:52:26 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory I, [2023-09-04T10:52:26.036541 #1] INFO -- : I, [2023-09-04T10:52:26.036767 #1] INFO -- : > rm -fr /shared/postgres_run/.s* I, [2023-09-04T10:52:26.040947 #1] INFO -- : I, [2023-09-04T10:52:26.041065 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid I, [2023-09-04T10:52:26.044903 #1] INFO -- : I, [2023-09-04T10:52:26.045086 #1] INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp I, [2023-09-04T10:52:26.049114 #1] INFO -- : I, [2023-09-04T10:52:26.049543 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp I, [2023-09-04T10:52:26.052949 #1] INFO -- : I, [2023-09-04T10:52:26.059452 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown: I, [2023-09-04T10:52:26.066126 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown: I, [2023-09-04T10:52:26.072688 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown: I, [2023-09-04T10:52:26.079432 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown: I, [2023-09-04T10:52:26.079705 #1] INFO -- : > chown -R root /var/lib/postgresql/13/main I, [2023-09-04T10:52:26.608052 #1] INFO -- : I, [2023-09-04T10:52:26.608287 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0 I, [2023-09-04T10:52:26.611255 #1] INFO -- : I, [2023-09-04T10:52:26.611303 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data I, [2023-09-04T10:52:26.629541 #1] INFO -- : I, [2023-09-04T10:52:26.629753 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql I, [2023-09-04T10:52:26.632565 #1] INFO -- : I, [2023-09-04T10:52:26.632795 #1] INFO -- : > /root/upgrade_postgres I, [2023-09-04T10:52:26.637276 #1] INFO -- : I, [2023-09-04T10:52:26.637383 #1] INFO -- : > rm /root/upgrade_postgres I, [2023-09-04T10:52:26.639574 #1] INFO -- : I, [2023-09-04T10:52:26.639907 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.640746 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.641188 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.641578 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.641927 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.642200 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.642443 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup I, [2023-09-04T10:52:26.645547 #1] INFO -- : I, [2023-09-04T10:52:26.645979 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.646319 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.646601 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf I, [2023-09-04T10:52:26.646881 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/13/main/pg_hba.conf I, [2023-09-04T10:52:26.647139 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2023-09-04T10:52:26.647405 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf I, [2023-09-04T10:52:26.647603 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main I, [2023-09-04T10:52:26.649431 #1] INFO -- : > sleep 5 2023-09-04 10:52:26.682 UTC [42] LOG: starting PostgreSQL 13.12 (Debian 13.12-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2023-09-04 10:52:26.682 UTC [42] LOG: listening on IPv4 address "0.0.0.0", port 5432 2023-09-04 10:52:26.682 UTC [42] LOG: listening on IPv6 address "::", port 5432 2023-09-04 10:52:26.683 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2023-09-04 10:52:26.687 UTC [45] LOG: database system was interrupted; last known up at 2023-09-03 21:07:49 UTC 2023-09-04 10:52:26.780 UTC [45] LOG: invalid resource manager ID in primary checkpoint record 2023-09-04 10:52:26.780 UTC [45] PANIC: could not locate a valid checkpoint record 2023-09-04 10:52:27.574 UTC [42] LOG: startup process (PID 45) was terminated by signal 6: Aborted 2023-09-04 10:52:27.574 UTC [42] LOG: aborting startup due to startup process failure 2023-09-04 10:52:27.632 UTC [42] LOG: database system is shut down I, [2023-09-04T10:52:31.652061 #1] INFO -- : I, [2023-09-04T10:52:31.652332 #1] INFO -- : > su postgres -c 'createdb discourse' || true createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory Is the server running locally and accepting connections on that socket? I, [2023-09-04T10:52:31.700646 #1] INFO -- : I, [2023-09-04T10:52:31.700881 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory Is the server running locally and accepting connections on that socket? I, [2023-09-04T10:52:31.748773 #1] INFO -- : I, [2023-09-04T10:52:31.749019 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory Is the server running locally and accepting connections on that socket? I, [2023-09-04T10:52:31.795035 #1] INFO -- : I, [2023-09-04T10:52:31.795255 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory Is the server running locally and accepting connections on that socket? I, [2023-09-04T10:52:31.839652 #1] INFO -- : I, [2023-09-04T10:52:31.840020 #1] INFO -- : Terminating async processes FAILED -------------------- Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return # Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn' exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'" bootstrap failed with exit code 2 ** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. d81222f4e7344503fa34eeb56f68542223c7e0dd4dff9dfbc59a3e0507996bb4

Thank you - I was afraid of that. Unfortunately the backup folder is empty… nothing the recover from there… :shushing_face: So either I try to recover whatever there is - either from the leftovers of the database and/or the Discourse content folders. Or start from scratch. :thinking: Any suggestions where to go from here? Assuming that some bits and pieces might be recovered? While the biggest portion is about a re-install?

Errors during rebuild app

Product Support Self-hosting

Airw0lf (Will) September 4, 2023, 7:34pm 6

A friend of mine was able to fix things.
Meaning there was a backup after all.
Once this was restored everything was/is back to normal.

Topic		Replies	Views
Bootstrap failed, please help :( Self-hosting	6	1108	January 3, 2023
Database system was not properly shut down error when rebuilding Self-hosting	3	7099	September 19, 2019
Postgres Errors on Rebuild Self-hosting	2	1761	September 16, 2015
Cannot rebuild app, Product Support	2	254	February 3, 2025
2.6.0 Beta4 update failed with what looks like db errors Self-hosting	6	827	October 19, 2020

Errors during rebuild app

Related topics