Errors during rebuild app

Airw0lf · September 4, 2023, 9:48am

Eaerlier this morning I tried to stop and rebuild the app but received several errors.

The stop didn’t report any errors.

The rebuild came with the following errors:

2023-09-04 08:30:19.937 UTC [45] LOG:  database system was interrupted; last known up at 2023-09-03 21:07:49 UTC
2023-09-04 08:30:20.045 UTC [45] LOG:  invalid resource manager ID in primary checkpoint record
2023-09-04 08:30:20.045 UTC [45] PANIC:  could not locate a valid checkpoint record
2023-09-04 08:30:20.859 UTC [42] LOG:  startup process (PID 45) was terminated by signal 6: Aborted
2023-09-04 08:30:20.859 UTC [42] LOG:  aborting startup due to startup process failure
2023-09-04 08:30:20.919 UTC [42] LOG:  database system is shut down

Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 55 exit 2>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.

Based on the logs it looks like it has something to do with this error?
PANIC: could not locate a valid checkpoint record

However, I’m not sure if this is really the case.
Let alone what to do about it.

Anyone suggestions on what could be wrong?
And ideally also a way to get this working (again)?

Thnak you,

JammyDodger · September 4, 2023, 10:11am

Could you scroll up and check for errors that could provide more detail? (or paste the output here so we can see)

Airw0lf · September 4, 2023, 10:54am

Below the output as a whole.
Is this what you where looking for?

root@forum:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20230825-1850: Pulling from discourse/base
Digest: sha256:3fd7032bbeb6f322ad7138ed154c0fc6177d75ebffba44871abea86620c0c7d5
Status: Image is up to date for discourse/base:2.0.20230825-1850
docker.io/discourse/base:2.0.20230825-1850
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-09-04T10:52:25.973226 #1]  INFO -- : Reading from stdin
I, [2023-09-04T10:52:25.978378 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-09-04T10:52:26.010346 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-09-04T10:52:26.010590 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-09-04T10:52:26.014888 #1]  INFO -- :
I, [2023-09-04T10:52:26.015219 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-09-04T10:52:26.018839 #1]  INFO -- :
I, [2023-09-04T10:52:26.019232 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-09-04T10:52:26.022489 #1]  INFO -- :
I, [2023-09-04T10:52:26.022851 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2023-09-04T10:52:26.025811 #1]  INFO -- :
I, [2023-09-04T10:52:26.026082 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2023-09-04T10:52:26.029384 #1]  INFO -- :
I, [2023-09-04T10:52:26.029659 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2023/09/04 10:52:26 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2023-09-04T10:52:26.036541 #1]  INFO -- :
I, [2023-09-04T10:52:26.036767 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2023-09-04T10:52:26.040947 #1]  INFO -- :
I, [2023-09-04T10:52:26.041065 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2023-09-04T10:52:26.044903 #1]  INFO -- :
I, [2023-09-04T10:52:26.045086 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-09-04T10:52:26.049114 #1]  INFO -- :
I, [2023-09-04T10:52:26.049543 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2023-09-04T10:52:26.052949 #1]  INFO -- :
I, [2023-09-04T10:52:26.059452 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown:
I, [2023-09-04T10:52:26.066126 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown:
I, [2023-09-04T10:52:26.072688 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown:
I, [2023-09-04T10:52:26.079432 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown:
I, [2023-09-04T10:52:26.079705 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2023-09-04T10:52:26.608052 #1]  INFO -- :
I, [2023-09-04T10:52:26.608287 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2023-09-04T10:52:26.611255 #1]  INFO -- :
I, [2023-09-04T10:52:26.611303 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2023-09-04T10:52:26.629541 #1]  INFO -- :
I, [2023-09-04T10:52:26.629753 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2023-09-04T10:52:26.632565 #1]  INFO -- :
I, [2023-09-04T10:52:26.632795 #1]  INFO -- : > /root/upgrade_postgres
I, [2023-09-04T10:52:26.637276 #1]  INFO -- :
I, [2023-09-04T10:52:26.637383 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2023-09-04T10:52:26.639574 #1]  INFO -- :
I, [2023-09-04T10:52:26.639907 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.640746 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.641188 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.641578 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.641927 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.642200 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.642443 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2023-09-04T10:52:26.645547 #1]  INFO -- :
I, [2023-09-04T10:52:26.645979 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.646319 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.646601 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2023-09-04T10:52:26.646881 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2023-09-04T10:52:26.647139 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-09-04T10:52:26.647405 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2023-09-04T10:52:26.647603 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2023-09-04T10:52:26.649431 #1]  INFO -- : > sleep 5
2023-09-04 10:52:26.682 UTC [42] LOG:  starting PostgreSQL 13.12 (Debian 13.12-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-09-04 10:52:26.682 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-09-04 10:52:26.682 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2023-09-04 10:52:26.683 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-09-04 10:52:26.687 UTC [45] LOG:  database system was interrupted; last known up at 2023-09-03 21:07:49 UTC
2023-09-04 10:52:26.780 UTC [45] LOG:  invalid resource manager ID in primary checkpoint record
2023-09-04 10:52:26.780 UTC [45] PANIC:  could not locate a valid checkpoint record
2023-09-04 10:52:27.574 UTC [42] LOG:  startup process (PID 45) was terminated by signal 6: Aborted
2023-09-04 10:52:27.574 UTC [42] LOG:  aborting startup due to startup process failure
2023-09-04 10:52:27.632 UTC [42] LOG:  database system is shut down
I, [2023-09-04T10:52:31.652061 #1]  INFO -- :
I, [2023-09-04T10:52:31.652332 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-09-04T10:52:31.700646 #1]  INFO -- :
I, [2023-09-04T10:52:31.700881 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-09-04T10:52:31.748773 #1]  INFO -- :
I, [2023-09-04T10:52:31.749019 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-09-04T10:52:31.795035 #1]  INFO -- :
I, [2023-09-04T10:52:31.795255 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
I, [2023-09-04T10:52:31.839652 #1]  INFO -- :
I, [2023-09-04T10:52:31.840020 #1]  INFO -- : Terminating async processes


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 55 exit 2>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
d81222f4e7344503fa34eeb56f68542223c7e0dd4dff9dfbc59a3e0507996bb4

Falco · September 4, 2023, 2:59pm

Will:

database system was interrupted; last known up at 2023-09-03 21:07:49 UTC
invalid resource manager ID in primary checkpoint record
could not locate a valid checkpoint record

database system is shut down

I’m afraid your database is corrupted. I’d try recovering from a recent backup.

Airw0lf · September 4, 2023, 3:46pm

Thank you - I was afraid of that.
Unfortunately the backup folder is empty… nothing the recover from there…

So either I try to recover whatever there is - either from the leftovers of the database and/or the Discourse content folders. Or start from scratch.

Any suggestions where to go from here?
Assuming that some bits and pieces might be recovered?
While the biggest portion is about a re-install?

Airw0lf · September 4, 2023, 7:34pm

A friend of mine was able to fix things.
Meaning there was a backup after all.
Once this was restored everything was/is back to normal.

system · October 4, 2023, 7:35pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Database system was not properly shut down error when rebuilding Installation	4	6714	September 19, 2019
Forum offline due to failed rebuilds on Tests-Pass Installation	1	895	December 11, 2020
Pups error on rebuild :dog: Installation	2	532	February 13, 2023
Upgrade Failed from 2.7.0.beta1 to 2.7.0.beta3 Installation	13	1864	February 15, 2021
PostgreSQL Error While Rebuilding Installation	11	1786	July 15, 2020

Errors during rebuild app

Related topics