Hanging on app rebuild

AI-generated summary

A user, heyrocker, is experiencing issues during a Discourse app rebuild on a new Droplet, with the process hanging and eventually failing with an error. The logs show a Redis background save completing successfully, followed by a Killed message and an error related to webpack. After troubleshooting, JammyDodger suggests the issue might be insufficient memory, specifically pointing to swap space. heyrocker confirms that the problem was indeed the lack of swap, which was resolved by setting it up.

I’m in the process of moving to a new Droplet using this guide

I’ve gotten to where I need to rebuild, and it goes along fine for a while before getting here:

110:M 21 Dec 2023 17:29:25.029 * Background saving started by pid 2933
2933:C 21 Dec 2023 17:29:25.583 * DB saved on disk
2933:C 21 Dec 2023 17:29:25.590 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
110:M 21 Dec 2023 17:29:25.632 * Background saving terminated with success

Looking at older posts which mention this issue, most of the suggestions center around rebooting your VPS and retrying the command. I did this and it did not resolve the problem. I have now run the rebuild three times with the following results:

  1. Reboot without it returning
  2. Waited about an hour and the following finally happened
110:M 21 Dec 2023 17:29:25.022 * 100 changes in 300 seconds. Saving...
110:M 21 Dec 2023 17:29:25.029 * Background saving started by pid 2933
2933:C 21 Dec 2023 17:29:25.583 * DB saved on disk
2933:C 21 Dec 2023 17:29:25.590 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
110:M 21 Dec 2023 17:29:25.632 * Background saving terminated with success
^[[HKilled
error Command failed with exit code 137.
I, [2023-12-21T18:19:23.241028 #1]  INFO -- : yarn run v1.22.19
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build
Building
Environment: development
WARNING: ember-test-selectors: You are using an unsupported ember-cli-babel version. data-test properties are not automatically stripped from your JS code.
building...
...[ConfigLoader]
...[Babel: discourse-widget-hbs > applyPatches]
...[Babel: ember-source > applyPatches]
...[ember.js]
...[Babel: discourse-common > applyPatches]
...[Babel: truth-helpers > applyPatches]
...[Babel: ember-tracked-storage-polyfill > applyPatches]
...[Babel: @ember/render-modifiers > applyPatches]
...[Babel: @ember/test-helpers > applyPatches]
...[Babel: @ember/test-waiters > applyPatches]
...[Babel: @embroider/util > applyPatches]
...[Babel: @glimmer/component > applyPatches]
...[Babel: dialog-holder > applyPatches]
...[Babel: ember-this-fallback > applyPatches]
...[Babel: ember-buffered-proxy > applyPatches]
...[Babel: ember-exam > applyPatches]
...[Babel: ember-functions-as-helper-polyfill > applyPatches]
...[Babel: ember-resize-observer-service > applyPatches]
...[Babel: float-kit > applyPatches]
...[Babel: select-kit > applyPatches]
...[@embroider/compat/app]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
...[@embroider/webpack]
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I, [2023-12-21T18:19:23.255055 #1]  INFO -- : Terminating async processes
I, [2023-12-21T18:19:23.255857 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
2023-12-21 18:19:23.256 UTC [41] LOG:  received fast shutdown request
I, [2023-12-21T18:19:23.257064 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 110
110:signal-handler (1703182763) Received SIGTERM scheduling shutdown...
2023-12-21 18:19:23.265 UTC [41] LOG:  aborting any active transactions
110:M 21 Dec 2023 18:19:23.269 # User requested shutdown...
110:M 21 Dec 2023 18:19:23.270 * Saving the final RDB snapshot before exiting.
2023-12-21 18:19:23.284 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2023-12-21 18:19:23.285 UTC [45] LOG:  shutting down
2023-12-21 18:19:23.375 UTC [41] LOG:  database system is shut down
110:M 21 Dec 2023 18:19:23.638 * DB saved on disk
110:M 21 Dec 2023 18:19:23.639 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 2840 exit 137>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 137
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
75e3ba35e779508ddbe9e536babf134fb2f100a50e05c38c1f26555d76801c98
  1. Ran for about an hour before I got a broken pipe error and disconnected

Any clues as to how to proceed from here?

This droplet is 2G RAM / 50G disk running Ubuntu 23.10 x64. The rsync command seemed to run as expected.

Thank you!

I think error 137 is not enough memory. Do you have enough swap set up?

Argh this was it, I had not set up swap. Thank you, all is well now.