Down after upgrade: yarn: not found

While trying to upgrade my self-hosted Discourse instance forum.embeetle.com:

git pull
./launcher rebuild app

I got this error:

sh: 1: yarn: not found

and the forum is down.

This issue looks similar (also mentions yarn: not found), but doesn’t provide a solution that I know how to apply:

Here is the last part of the output of ./launcher rebuild app, including the error:

I, [2022-04-15T12:58:22.640335 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake plugin:pull_compatible_all'
WARNING: Plugins were activated before running `rake plugin:pull_compatible_all`
  You should prefix this command with LOAD_PLUGINS=0
I, [2022-04-15T12:58:28.725935 #1]  INFO -- : discourse-bbcode-color is already at latest compatible version
docker_manager is already at latest compatible version
discourse-math is already at latest compatible version

I, [2022-04-15T12:58:28.726300 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
I, [2022-04-15T12:58:35.125132 #1]  INFO -- : 
I, [2022-04-15T12:58:35.125478 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
sh: 1: yarn: not found
I, [2022-04-15T12:58:39.396023 #1]  INFO -- : Checking 'Tag Icons' for 'default'... 
I, [2022-04-15T12:58:39.396636 #1]  INFO -- : Terminating async processes
I, [2022-04-15T12:58:39.396757 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 57
I, [2022-04-15T12:58:39.396823 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 174
2022-04-15 12:58:39.396 UTC [57] LOG:  received fast shutdown request
174:signal-handler (1650027519) Received SIGTERM scheduling shutdown...
174:M 15 Apr 2022 12:58:39.399 # User requested shutdown...
174:M 15 Apr 2022 12:58:39.399 * Saving the final RDB snapshot before exiting.
2022-04-15 12:58:39.416 UTC [57] LOG:  aborting any active transactions
174:M 15 Apr 2022 12:58:39.417 * DB saved on disk
174:M 15 Apr 2022 12:58:39.417 # Redis is now ready to exit, bye bye...
2022-04-15 12:58:39.418 UTC [57] LOG:  background worker "logical replication launcher" (PID 66) exited with exit code 1
2022-04-15 12:58:39.421 UTC [61] LOG:  shutting down
2022-04-15 12:58:39.544 UTC [57] LOG:  database system is shut down

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 5379 exit 127>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
13c38843d2b7e10b51c38f0d998f9e6e1c84fec93c7fc2ab2c044834ee6a8e59
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

The output of discourse-doctor is too large to post here, so I include the part that looks most significant to me:

root@foechoer:/srv/embeetle_forum/discourse# ./discourse-doctor
DISCOURSE DOCTOR Fri Apr 15 14:59:02 CEST 2022
OS: Linux foechoer.sikando.com 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


Found containers/app.yml

==================== YML SETTINGS ====================
DISCOURSE_HOSTNAME=forum.embeetle.com
SMTP_ADDRESS=smtp.zoho.com #mail.embeetle.com
DEVELOPER_EMAILS=johan@embeetle.com,kristof@embeetle.com,matic@embeetle.com
SMTP_PASSWORD=PASSWORD
SMTP_PORT=587
SMTP_USER_NAME=forum@embedeer.com
LETSENCRYPT_ACCOUNT_EMAIL=postmaster@sikando.com

==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 20.10.14, build a224086

DOCKER PROCESSES (docker ps -a)

CONTAINER ID   IMAGE                              COMMAND                  CREATED         STATUS                     PORTS      NAMES
f972c6ac40db   nginx:latest                       "/docker-entrypoint.…"   16 hours ago    Up 16 hours                80/tcp     docker_chipselect_1
b8eaa7fc632d   docker_cs_php                      "docker-php-entrypoi…"   16 hours ago    Up 16 hours                9000/tcp   docker_cs_php_1
51c18c3af7d2   mariadb:latest                     "docker-entrypoint.s…"   16 hours ago    Up 16 hours                3306/tcp   docker_cs_mysql_1
b018a429fc83   hello-world                        "/hello"                 3 days ago      Exited (0) 3 days ago                 blissful_wiles
f4064f10924a   local_discourse/app                "/sbin/boot"             11 months ago   Exited (5) 2 minutes ago              app
bff8181744a2   discourse/base:2.0.20210415-1332   "/bin/bash -c 'cd /p…"   11 months ago   Exited (0) 11 months ago              vigilant_einstein
fb6cca53886d   discourse/base:2.0.20210415-1332   "/bin/bash -c 'cd /p…"   11 months ago   Exited (0) 11 months ago              gifted_mclaren
ea974bead75f   discourse/base:2.0.20201221-2020   "/bin/bash -c 'cd /p…"   13 months ago   Exited (0) 13 months ago              ecstatic_margulis

==================== SERIOUS PROBLEM!!!! ====================
app not running!
Attempting to rebuild
==================== REBUILD LOG ====================

...(omitted long output)...

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 5379 exit 127>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
1fc0ce85c275da0c2c9f39bc496581b4a5a80f0888378733e1e6e177698dc110
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
==================== END REBUILD LOG ====================
Failed to rebuild app.

Checking your domain name . . .

Connection to forum.embeetle.com succeeded.
You should probably remove any non-standard plugins and rebuild.
Attempting to restart existing container. . . 

starting up existing container
+ /usr/bin/docker start app
app
f4064f10924a   local_discourse/app   "/sbin/boot"             11 months ago   Up Less than a second              app
Restarted the container.


==================== PLUGINS ====================
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-math.git
          - git clone https://github.com/discourse/discourse-bbcode-color.git

No non-official plugins detected.

See https://github.com/discourse/discourse/blob/master/lib/plugin/metadata.rb for the official list.

========================================
Discourse version at forum.embeetle.com: NOT FOUND
Discourse version at localhost: NOT FOUND


==================== MEMORY INFORMATION ====================
RAM (MB): 20553

              total        used        free      shared  buff/cache   available
Mem:          20071        4444        9431         292        6195       15517
Swap:          2047          96        1951

==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
df: /var/discourse: No such file or directory
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       1.4T  1.1T  279G  79% /

==================== DISK INFORMATION ====================
Disk /dev/loop0: 111.6 MiB, 117014528 bytes, 228544 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 61.9 MiB, 64901120 bytes, 126760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 43.9 MiB, 45998080 bytes, 89840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 110.6 MiB, 115986432 bytes, 226536 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 61.9 MiB, 64901120 bytes, 126760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop6: 43.9 MiB, 45998080 bytes, 89840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 1.4 TiB, 1503238553600 bytes, 2936012800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x645c17b2

Device     Boot   Start        End    Sectors  Size Id Type
/dev/sda1  *       2048    1953791    1951744  953M 83 Linux
/dev/sda2       1953792 2936010751 2934056960  1.4T 83 Linux

==================== END DISK INFORMATION ====================

==================== MAIL TEST ====================
For a robust test, get an address from http://www.mail-tester.com/
Or just send a test message to yourself.
Email address for mail test? ('n' to skip) [johan@embeetle.com]: 
Sending mail to johan@embeetle.com. . . 
Testing sending to johan@embeetle.com using smtp.zoho.com:587, username:forum@embedeer.com with plain auth.
SMTP server connection successful.
Sending to johan@embeetle.com. . . 
Mail accepted by SMTP server.
Message-ID: 0268f0a5-e5d3-430d-85ab-3c7f3e173487@forum.embeetle.com

If you do not receive the message, check your SPAM folder
or test again using a service like http://www.mail-tester.com/.

If the message is not delivered it is not a problem with Discourse.
Check the SMTP server logs for the above Message ID to see why it
failed to deliver the message.
Replacing: SMTP_PASSWORD
Replacing: LETSENCRYPT_ACCOUNT_EMAIL
Replacing: DEVELOPER_EMAILS
Replacing: DISCOURSE_DB_PASSWORD
Replacing: Sending mail to

==================== DONE! ====================
Would you like to serve a publicly available version of this file? (Y/n)y
Publicly available log not generated.

You can examine the output of this script with 
LESS=-Ri less /tmp/discourse-debug.txt

BUT FIRST, make sure that you know the first three commands below!!!

Commands to know when viewing the file with the above command (called 'less'): 
q              -- quit
/error<ENTER>  -- search for the word 'error'
n              -- search for the next occurrence
g              -- go to the beginning of the file
f              -- go forward a page
b              -- go back a page
G              -- go to the end of the file

I believe the error in the other topic was down to not using the correct branch of discourse (master was renamed to main). The installer should automatically swop it over, but there have been a few cases where it hasn’t.

From what I can glean from a search, you could try:

cd /var/discourse
git checkout main

And then try rebuilding?

(I’ll throw a bat signal up for @pfaffman and @Falco too as they have a lot more experience of this than I do :slight_smile: )

1 Like

The forum is up again.

This is what I did:

  1. Use git status to check the current branch. It is already main, but not the latest revision, in spite of the fact that I pulled a few hours ago. No idea why. It was stuck at revision 990519e237 “Rename master to main” of 2021-08-03 20:50:04.

  2. Did git pull again. This moved to revision 2a9faf7e56 “Bump default base image” of 2022-04-13 06:22:53

  3. Did ./launcher rebuild app. This started writing some extremely long lines (looking like json) to my terminal, after which the connection to my server was closed.

  4. Logged back into the server and did ./launcher start app. No errors, but the forum was still not online.

  5. Tried again ./launcher rebuild app. This worked without issues and the forum is back online.

I am happy, but still not sure what went wrong. Probably related to the branch rename from master to main, as you @JammyDodger guessed.

2 Likes