My server runs on in a Virtual Machine hosted by one of the major cloud providers.
I successfully installed discourse on it and it’s been running fine for the last month.
Today, I decided to change the specifications of my VM back to its original configuration(*) and rebooted. Upon startup, whilst everything else on my server is running fine, I’m getting a 502 Bad Gateway error when I attempt to access the discourse forum. Thinking the docker instance had not automatically started up, I SSHed into my server and ran ./launcher start app but got a message saying I had insufficient space remaining (5GB available), so I ran df -h which tells me I actually have 14GB available. So I ran ./launcher start app again, but this time I got a warning that docker was going to download stuff and to be patient. After some processing, I got the message Nothing to do, your container has already started!. However, my attempts to access the forum still returned 502 Bad Gateway.
After consulting this forum here, I decided to run ./launcher rebuild app and got the following errors, something to do with PostgreSQL:
user@host:[16:48]:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 60 app
app
cd /pups && git pull && /pups/bin/pups --stdin
Already up to date.
I, [2020-07-01T07:19:42.821347 #1] INFO -- : Loading --stdin
I, [2020-07-01T07:19:42.831806 #1] INFO -- : > locale-gen $LANG && update-locale
I, [2020-07-01T07:19:42.879007 #1] INFO -- : Generating locales (this might take a while)...
Generation complete.
I, [2020-07-01T07:19:42.879431 #1] INFO -- : > mkdir -p /shared/postgres_run
I, [2020-07-01T07:19:42.885054 #1] INFO -- :
I, [2020-07-01T07:19:42.885734 #1] INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2020-07-01T07:19:42.891655 #1] INFO -- :
I, [2020-07-01T07:19:42.892269 #1] INFO -- : > chmod 775 /shared/postgres_run
I, [2020-07-01T07:19:42.898103 #1] INFO -- :
I, [2020-07-01T07:19:42.898942 #1] INFO -- : > rm -fr /var/run/postgresql
I, [2020-07-01T07:19:42.905607 #1] INFO -- :
I, [2020-07-01T07:19:42.906463 #1] INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2020-07-01T07:19:42.912617 #1] INFO -- :
I, [2020-07-01T07:19:42.913233 #1] INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2020/07/01 07:19:42 socat[26] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): No such file or directory
I, [2020-07-01T07:19:42.925688 #1] INFO -- :
I, [2020-07-01T07:19:42.926081 #1] INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2020-07-01T07:19:42.931174 #1] INFO -- :
I, [2020-07-01T07:19:42.931649 #1] INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2020-07-01T07:19:42.938152 #1] INFO -- :
I, [2020-07-01T07:19:42.938850 #1] INFO -- : > mkdir -p /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-07-01T07:19:42.943575 #1] INFO -- :
I, [2020-07-01T07:19:42.944331 #1] INFO -- : > chown postgres:postgres /shared/postgres_run/12-main.pg_stat_tmp
I, [2020-07-01T07:19:42.949159 #1] INFO -- :
I, [2020-07-01T07:19:42.961190 #1] INFO -- : File > /etc/service/postgres/run chmod: +x chown:
I, [2020-07-01T07:19:42.973345 #1] INFO -- : File > /etc/service/postgres/log/run chmod: +x chown:
I, [2020-07-01T07:19:42.983929 #1] INFO -- : File > /etc/runit/3.d/99-postgres chmod: +x chown:
I, [2020-07-01T07:19:42.994843 #1] INFO -- : File > /root/upgrade_postgres chmod: +x chown:
I, [2020-07-01T07:19:42.995487 #1] INFO -- : > chown -R root /var/lib/postgresql/12/main
I, [2020-07-01T07:19:44.012812 #1] INFO -- :
I, [2020-07-01T07:19:44.013656 #1] INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/12/bin/initdb -D /shared/postgres_data || exit 0
I, [2020-07-01T07:19:44.019545 #1] INFO -- :
I, [2020-07-01T07:19:44.019872 #1] INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2020-07-01T07:19:44.064432 #1] INFO -- :
I, [2020-07-01T07:19:44.065186 #1] INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2020-07-01T07:19:44.071385 #1] INFO -- :
I, [2020-07-01T07:19:44.072196 #1] INFO -- : > /root/upgrade_postgres
I, [2020-07-01T07:19:44.084004 #1] INFO -- :
I, [2020-07-01T07:19:44.084662 #1] INFO -- : > rm /root/upgrade_postgres
I, [2020-07-01T07:19:44.090399 #1] INFO -- :
I, [2020-07-01T07:19:44.092280 #1] INFO -- : Replacing data_directory = '/var/lib/postgresql/12/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.093969 #1] INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.095204 #1] INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.095937 #1] INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.096695 #1] INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.097554 #1] INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.101971 #1] INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2020-07-01T07:19:44.112672 #1] INFO -- :
I, [2020-07-01T07:19:44.113831 #1] INFO -- : Replacing (?-mix:#?max_wal_senders *=.*) with max_wal_senders = $db_max_wal_senders in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.114973 #1] INFO -- : Replacing (?-mix:#?wal_level *=.*) with wal_level = $db_wal_level in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.116047 #1] INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.117033 #1] INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.118051 #1] INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/12/main/postgresql.conf
I, [2020-07-01T07:19:44.119352 #1] INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres peer in /etc/postgresql/12/main/pg_hba.conf
I, [2020-07-01T07:19:44.120299 #1] INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/12/main/pg_hba.conf
I, [2020-07-01T07:19:44.121038 #1] INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/12/bin/postmaster -D /etc/postgresql/12/main
I, [2020-07-01T07:19:44.126334 #1] INFO -- : > sleep 5
2020-07-01 07:19:44.157 UTC [49] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-07-01 07:19:44.158 UTC [49] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-07-01 07:19:44.158 UTC [49] LOG: listening on IPv6 address "::", port 5432
2020-07-01 07:19:44.161 UTC [49] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-07-01 07:19:44.162 UTC [49] FATAL: could not map anonymous shared memory: Cannot allocate memory
2020-07-01 07:19:44.162 UTC [49] HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 4423172096 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2020-07-01 07:19:44.162 UTC [49] LOG: database system is shut down
I, [2020-07-01T07:19:49.141762 #1] INFO -- :
I, [2020-07-01T07:19:49.142221 #1] INFO -- : > su postgres -c 'createdb discourse' || true
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-01T07:19:49.227852 #1] INFO -- :
I, [2020-07-01T07:19:49.228226 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-01T07:19:49.330486 #1] INFO -- :
I, [2020-07-01T07:19:49.330822 #1] INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-01T07:19:49.425970 #1] INFO -- :
I, [2020-07-01T07:19:49.426356 #1] INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I, [2020-07-01T07:19:49.506638 #1] INFO -- :
I, [2020-07-01T07:19:49.507202 #1] INFO -- : Terminating async processes
FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 75 exit 2>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
eb41679f76cd749ccd8c84a7543365d093619b80df6fc6750b9349fb63565fa1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
user@host:[17:19]:/var/discourse#
Strangely, in spite of the errors above, running ./launcher start app produces no errors:
starting up existing container
+ /usr/bin/docker start app
app
With the instance running, I tried using ./launcher enter app to get into the container. (In my humble opinion the available tools in the container are very poor (yes I’m a nano user and like to have various aliases mapped; e.g. ll). I’m not able to find the physical path to the folders within the docker instance (as I would like to download them using a FTP client).
In /var/log/nginx/error.log I have the following error entry for each time I refresh my browser:
2020/07/01 07:44:16 [error] 646#646: *3 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xx.0.1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "discourse.myDomain.com"
What could be the cause of my problem? Why is Postgre suddenly not functioning?
(*) A week after installing Discourse, I upgraded my server with more CPUs and memory. I needed to do this to run a video conference I hosted. With the conference done, I’ve gone back to my normal configuration. Note that I did not change the disk sizes at any point during the specification changes.
