Broke my installation with a failed upgrade - how can I recover?

TLDR: I clicked the update button in my Discourse admin panel, and the update failed. I decided to reboot the whole thing and try the update process again. Unfortunately my Discourse admin panel doesn’t come up. I’d like to get things working again.

Supporting info: This is in a DigitalOcean VM that serves a WordPress site. I set it up years ago and forgot many details, but by some magic Nginx looks at the requested URL and decides whether to activate WP for the main content or Discourse for the conversations. The WP side still works.

Given the naive and casual approach suggested in my intro, it will be no surprise that I can’t pinpoint the exact nature of the installation error that landed me in this sad state. I don’t even know where to find the log files that might contain clues about getting the upgrade process back on track.

Any hints would be most welcome. I am not experienced in web admin lore, but I am comfortable with the command line on Debian-based systems. The software just mentioned is all running on Ubuntu 20.04.6 LTS, and I have SSH access to the relevant platform. Thanks!

hi, try to rebuild the container

1 Like

You can try

cd /var/discourse
./launcher rebuild app

That might do the trick. If not, it’ll likely be hard to guess.

2 Likes

Thanks a lot for your replies. I will follow Jay’s suggestion and report back. (But it may be several hours before I do. First I have to make pizza and feed it to my guests. :slight_smile:

1 Like

Unfortunately my DigitalOcean VM appears to be too small. If I say du -sh after a fresh login, it says my assigned disk /dev/vda1 has about 7 GiB free. After I give the launcher rebuild app command, a bunch of downloading and unpacking happens and then the script aborts, saying that it sees only 3.3G available and that it refuses to continue unless there is 5G or more to use.

Of course I have done what I can to locate and delete extraneous stuff, but finding another 1.7G to purge will be a tall order. Are there any expert tips for this situation? (Either in the category of huge targets ripe for deletion, or in the realm of asking the launcher to be less demanding.)

PS: Alternatively, can I rebuild the app on some stronger machine and then just upload a suitable file to this underpowered little VM? Or is this excessively arcane black magic?

Try to execute the command ./launcher cleanup. This operation can assist you in purging the storage space consumed by Docker.

Additionally, if you are certain that you previously followed the official installation guide, you can execute ./launcher destroy app to remove the previous broken container. However, it is imperative to ensure that your database and uploads are stored on the host file system rather than within the container itself.

Indeed, you can perform the rebuild on a separate machine and subsequently transfer the Docker image to your VM. However, you’ll need to manually execute rake db:migrate command on your VM. I’m not entirely certain whether this approach might introduce other issues.

1 Like

Just migrate to a new VPS with more room.

Takes about 30 mins if that.

2 Likes

I’m not 100% certain, but I’d like to try this. How can I do the imperative step of preserving my database and uploads? Would it suffice to copy the files in the directory /var/discourse/shared/standalone to some safe location before doing anything serious? (Is there a pointer to a standard process for the kind of hands-on restoration project this would kick off?)

I think @Robert’s pragmatic response would be the best choice for most people in a similar situation. I’m taking a different direction because my context is unique in several ways. Of these, here is the biggest: my installation is a small-scale hobby project where losing everything would be disappointing but not devastating.

Thanks again.

If stuff is in the shared/standalone directory, you’re safe.

If you’ve got wordpress and discourse you probably need to upgrade to a 2gb/50gb vm. If you’ve deleted all the backups and done a launcher cleanup then you likely have some all you can (oh, you can also clear the apt cache if you Google that).

1 Like

May or may not be the same problem we just encountered: We were using maxmind’s API to do the ip>geolocation conversion. The old version required the maxmind API key in the app.yml file. Recent (undocumented?) change requires the maxmind api key AND username in the app.yml. Lacking the username causes the forum to fail to build.

2 Likes

Thanks, @Frully, but I think this is a different issue. Here is the error message that stops me:

Status: Downloaded newer image for discourse/base:2.0.20240602-0023                                                                                                                                                                                        
docker.io/discourse/base:2.0.20240602-0023                                                                                                                                                                                                                 
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue                                                                                                                                    
Filesystem      Size  Used Avail Use% Mounted on                                                                                                                                                                                                           
/dev/vda1        25G   21G  3.5G  86% /

I’ll continue my troubleshooting adventure tonight after work.

Friends, I’m going to change tactics and pursue @merefield 's idea of migrating my whole setup to a bigger VPS. Figuring out how to transfer my assets from the broken setup into the new working one will be a challenging piece of this activity, for sure, but I will try a few common-sense methods and start a new thread if none of them work. With thanks to all who chipped in, I think it’s safe to call this thread closed.

4 Likes

Epilog: The process of moving to a new host taught me something useful about running shell commands right inside the Discourse container, now described in another thread. There is a little more information about saving disk space in that other thread as well.

2 Likes