# Discourse upgrade via Web UI Fails & SSH Upgrade Brings Down Discourse Instance **URL:** \ **Category:** Self-hosting\ **Created:** [November 25, 2021, 4:49pm UTC](https://meta.discourse.org/t/discourse-upgrade-via-web-ui-fails-ssh-upgrade-brings-down-discourse-instance/210164 "2021-11-25T16:49:42Z")\ **Posts on this page:** 1\ **Showing post:** 1
**Author:** ![billmaya](https://avatars.discourse-cdn.com/v4/letter/b/e495f1/32.png) [@billmaya](https://meta.discourse.org/u/billmaya)\ **Post date:** [November 25, 2021, 4:49pm UTC](https://meta.discourse.org/t/discourse-upgrade-via-web-ui-fails-ssh-upgrade-brings-down-discourse-instance/210164/1 "2021-11-25T16:49:42Z")
NOTE: Original post updated 11/25/21 PM EST with new information Notified of critical security updates to my Discourse installation I attempted to update my installation using the Web UI (/admin/upgrade) as I have done in the past. There were two pieces of software that needed to be upgraded—Docker Manager and Discourse. The Docker Manager had to be upgraded first (the Discourse upgrade button was disabled). I started the Docker Manager upgrade using the Web UI and it completed successfully. I then started the Discourse upgrade but it failed midway through. When I refreshed the Web UI I saw the following message: ![image](https://global.discourse-cdn.com/meta/original/3X/3/d/3dcc0bb2f1e8a05b701a8ae36e5ccd2f4b72c79a.png) So, following the onscreen instructions, I SSH’d into the server, did a `git pull` and then ran `sudo ./launcher rebuild app` from the command line. The process finished but failed with a FAILED TO BOOTSTRAP error message. Here is the output of two runs of `sudo ./launcher rebuild app` at different times: - [launcher-rebuild-app-output-0.txt](https://gist.github.com/legably/3b905ab13fdffd8ad00c0d2f3d5ae486) - Lines 88-95 - [launcher-rebuild-app-output-1.txt](https://gist.github.com/legably/1d1e4cc7a3632dbbe3299324225f253a) - Lines 100-107 The line numbers after each file are where the only ERRORs appear. Both appear to be database and role related (the difference between both ranges is because the second attempted a `git pull` from the `discourse/base` repository). ```plaintext 2021-11-25 21:21:38.451 UTC [64] postgres@postgres ERROR: database "discourse" already exists 2021-11-25 21:21:38.451 UTC [64] postgres@postgres STATEMENT: CREATE DATABASE discourse; createdb: error: database creation failed: ERROR: database "discourse" already exists I, [2021-11-25T21:21:38.454429 #1] INFO -- : I, [2021-11-25T21:21:38.454908 #1] INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true 2021-11-25 21:21:38.531 UTC [68] postgres@discourse ERROR: role "discourse" already exists 2021-11-25 21:21:38.531 UTC [68] postgres@discourse STATEMENT: create user discourse; ERROR: role "discourse" already exists ``` This appears to dovetail with the FAILED error message displayed at the bottom of each Launcher Rebuild attempt. ```plaintext FAILED -------------------- Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return # Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn' exec failed with the params {"cd"=>"$home", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]} 13bbdd52e0835ba9dfddc5c367d63b6087a16553c3a77d27ca307734d6e16907 **FAILED TO BOOTSTRAP** please scroll up and look for earlier error messages, there may be more than one. ./discourse-doctor may help diagnose the problem. ``` _Note: These ERRORS are not the root problem. See “Solution” below._ Some people below have said that there is an issue with `redis` that is preventing a successful rebuild. I ran `sudo ./discourse-doctor at various times during the day. Here is the output from two of the runs: - [discourse-doctor-output-0.txt](https://gist.github.com/legably/3b905ab13fdffd8ad00c0d2f3d5ae486) - could not find ‘app’ container running; attempted rebuild but failed to restart the container - [discourse-doctor-output-1.txt](https://gist.github.com/legably/3aebb590951a3a1583b27869b073e757) - ‘app’ started manually with `sudo usr/bin/docker start app` before running `discourse-doctor` I verified that my Docker installation was running correctly by running `sudo docker run -it --rm hello-world` ```plaintext Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 2db29710123e: Pull complete Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. ``` I ran `sudo ./launcher cleanup` to make sure I had enough disk space. ```plaintext WARNING! This will remove all images without at least one container associated to them. Are you sure you want to continue? [y/N] y Deleted Images:
Total reclaimed space: 3.836GB $ df -hT /dev/xvda1 Filesystem Type Size Used Avail Use% Mounted on /dev/xvda1 ext4 30G 9.1G 20G 32% / ``` And I even checked my memory settings. ```plaintext $ free -h total used free shared buff/cache available Mem: 1.9G 304M 633M 20M 1.0G 1.5G Swap: 2.0G 0B 2.0G ``` A reboot of the server did not solve the issue but I did notice something interesting after rebooting the server. The Docker `app` container is running after a reboot. ```plaintext $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6449ec0061a0 local_discourse/app "/sbin/boot" 7 weeks ago Up 25 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp app ``` But when I go to the site I get a “502 Bad Gateway” error. ![image](https://global.discourse-cdn.com/meta/original/3X/d/e/de7cafd64aa871928c337a5c0c4cf1794faa35c1.png) When I stop the `app` container and go to the site I get a “Unable To Connect” error (which seems right since the container isn’t running). ![image](https://global.discourse-cdn.com/meta/original/3X/7/1/7122f199de1eea04ad0ee4a316002dabe7b09f22.png) But this puzzles me since I don’t have `Nginx` installed on this server. I can see in the Rebuild output where the process is copying Nginx files from one location to another but I cannot find the corresponding directories or files, specifically `nginx.conf` on my server anywhere. Ubuntu, Docker, and Discourse are not my primary skills but I am assuming that these files are being copied “within” the Docker `app` container. Thanks in advance; appreciate any additional help or direction with this issue, which seems to surface from time to time during Discourse upgrades. UPDATE: It turns out my assumption regarding the Docker `app` container having its own internal filesystem is correct. You can create a snapshot of the container filesystem and explore this filesystem using bash. ```plaintext # create image (snapshot) from container filesystem $ sudo docker commit mysnapshot $ sudo docker run -t -i mysnapshot /bin/bash ``` In the `app` filesystem there is an `nginx` directory that contains a Discourse configuration file. ```plaintext root@f91826d986eb:/etc/nginx/conf.d# ls -l total 12 -rw-r--r-- 1 root root 10568 Oct 3 21:33 discourse.conf ``` --- _[View the full topic](https://meta.discourse.org/t/discourse-upgrade-via-web-ui-fails-ssh-upgrade-brings-down-discourse-instance/210164)._