Failing to bootstrap

This is on a test machine. I was previously running discourse there - I messed up the installation and wasn’t able to update to the latest version which I thought was my mistake. After removing the whole discourse directory and clearing up docker I tried doing a completely fresh install before importing a backup from the live db.

Weirdly I’m still seeing the same issues which I am not able to resolve.

Here’s the failure output. Tried discourse-doctor already but didn’t bring up anything helpful.

...
I, [2022-06-04T18:42:29.087446 #1]  INFO -- : Terminating async processes
I, [2022-06-04T18:42:29.087672 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2022-06-04T18:42:29.087881 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
2022-06-04 18:42:29.088 UTC [42] LOG:  received fast shutdown request
103:signal-handler (1654368149) Received SIGTERM scheduling shutdown...
2022-06-04 18:42:29.118 UTC [42] LOG:  aborting any active transactions
2022-06-04 18:42:29.123 UTC [42] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
2022-06-04 18:42:29.123 UTC [46] LOG:  shutting down
103:M 04 Jun 2022 18:42:29.154 # User requested shutdown...
103:M 04 Jun 2022 18:42:29.154 * Saving the final RDB snapshot before exiting.
103:M 04 Jun 2022 18:42:29.159 * DB saved on disk
103:M 04 Jun 2022 18:42:29.159 # Redis is now ready to exit, bye bye...
2022-06-04 18:42:29.201 UTC [42] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 1102 exit 1>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
69cb25658efb6f16e4479bb98a2d0278d72e56028865730841ac1efacc5b8d9d
==================== END REBUILD LOG ====================

The server itself should be fine - plenty of disk space, enough resources otherwise. Any idea?

We need to see more of the log, I think, and in particular to see what happened with that rake command.

Please provide the outputs of

free
df

It might also be relevant if your log includes
WARNING overcommit_memory is set to 0!

1 Like

It does:

103:M 04 Jun 2022 18:40:07.369 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

here’s the rest you requested:

root@testserver-2021:/var/discourse# free
              total        used        free      shared  buff/cache   available
Mem:       16005396      353200    13366988        1100     2285208    15316292
Swap:             0           0           0
root@testserver-2021:/var/discourse# free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       343Mi        12Gi       1.0Mi       2.2Gi        14Gi
Swap:            0B          0B          0B
root@testserver-2021:/var/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.7G     0  7.7G   0% /dev
tmpfs           1.6G  1.1M  1.6G   1% /run
/dev/sda1       226G   44G  173G  21% /
tmpfs           7.7G     0  7.7G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/sda15       61M  5.2M   55M   9% /boot/efi
overlay         226G   44G  173G  21% /var/lib/docker/overlay2/c9457cf1821fb558a92c79e55bd6a70153b8ae0388732aa5eef17237b6924c25/merged
overlay         226G   44G  173G  21% /var/lib/docker/overlay2/6d350b54871378d4b0e7ac5d30e236df3bdd1c45cd3b5fb2e9ab67ffe7a1bba1/merged
tmpfs           1.6G     0  1.6G   0% /run/user/0
overlay         226G   44G  173G  21% /var/lib/docker/overlay2/104be98cc5c33e73e4119d2186b6d9d08123ffc79df872f48425e94a66ca6749/merged

I guess the overcommit messages is due to swap being 0? weirdly that has never changed since this servers exists…

Hmm… 16G RAM is quite a lot, so you might think you don’t need swap. But I would say it might not do any harm to add some. Without seeing your log I can’t say the problem is memory scarcity. But if it is, setting the overcommit mode might help, whether or not you have swap.

You might also try
dmesg | egrep -i "oom|memory"
to see if any processes were killed due to an out of memory condition.

But again, seeing more of your log is likely to help.

Edit: oops, added the -i

1 Like

I did create a swap and also set the overcommit mode. Nothing changed unfortunately.

Here’s the whole log from building the app https://pastebin.com/raw/R2B8Wneu

I see some redis failures saying port is in use but I can’t see where this comes from

root@testserver-2021:~# sudo lsof -i -P -n | grep LISTEN
systemd       1            root   89u  IPv6  15961      0t0  TCP *:9090 (LISTEN)
systemd-r   573 systemd-resolve   13u  IPv4  10890      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd        676            root    3u  IPv4  20387      0t0  TCP *:22 (LISTEN)
sshd        676            root    4u  IPv6  20389      0t0  TCP *:22 (LISTEN)
docker-pr   913            root    4u  IPv4  23665      0t0  TCP *:9100 (LISTEN)
docker-pr   921            root    4u  IPv6  21827      0t0  TCP *:9100 (LISTEN)
docker-pr   981            root    4u  IPv4  24732      0t0  TCP *:3003 (LISTEN)
docker-pr   989            root    4u  IPv6  22636      0t0  TCP *:3003 (LISTEN)
monitorix  1284       monitorix    3u  IPv4  27978      0t0  TCP *:8080 (LISTEN)

Thanks for the log - it looks like an S3 setup problem (which isn’t something I can help with, but I’m sure someone can.)

I, [2022-06-05T09:06:10.144445 #1] INFO – : > cd /var/www/discourse && su discourse -c ‘bundle exec rake db:migrate’
rake aborted!
Discourse::SiteSettingMissing: s3_upload_bucket
/var/www/discourse/lib/file_store/s3_store.rb:267:in `s3_bucket’

good find, Ed. Thanks. Looks like s3_bucket at some point changed to s3_upload_bucket and I do have those in containers/app.yml which seems to have caused the issue. At least building went fine now after I changed DISCOURSE_S3_BUCKET there to DISCOURSE_S3_UPLOAD_BUCKET.

I wish such changes would also introduce a check in the build process to avoid running into this - and good luck we always test our updates on a test machine.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.