Switching domain provider and also forum domain so better upgrade first… bad idea, forgot that 50% of my discourse upgrades fail .
Maybe a problem with “gem ‘mini_racer’” and found some too old distro/gcc comments but with 9 days before the domain expires I just decided to restore the backup, via console since the site is now down (and I just put out the message the forum url will change so people might actually try right now, uh oh).
Which also fails. Uh oh again, apparently site will be down a bit longer…
Logs attached, pretty clueless (as always), have some errands to run so just crossing my fingers for a simple solution.
For now, I just want to go back to the previous version.
Then, with enough time, I’ll check if it’s even possible to upgrade “easily” or if I finally after 10 years do a new Ubuntu from scratch and move stuff there. But remembering what ELSE I have done over the years… uh oh.
Yeah, but setting up a new server and considering whether to move the source control as well and whatever I have installed over the years to have in a 24/7 cloud droplet… Something I prefer to have at least a day to do. Which I don’t have now.
Ok, for now I’m just restoring last nights droplet and dropping the backup on that. I hope that goes well at least.
Seriously, if an upgrade fails there should be an automatic failsafe return to the previous state. Taking a backup of your droplet seems a bit of overkill and is dependent on the admin. Being paranoid and having two instances and always start by moving the active to the oldest and upgrading that… well, probably what I should do these days.
Well, had to update app.yml because of the domain switch and did app rebuild instead of restart by mistake without taking a snapshot in Digital Ocean first. So kinda forced myself into upgrading right now .
Which eventually worked. Since I hit a few snags (laughs allowed) I’ll publish my path of (mis)fortune here. Maybe someone stumbles on the exact error message and gets help from this.
Time taken 3.5h (guess I should have been lazy and just done the Digital Ocean 1 click setup but was curious if upgrading would work).
*****Upgrade Ubuntu 16.x → Ubuntu 18.x
*****Upgrade Ubuntu 18.x → Ubuntu 20.x (surprised at how well this actually worked, way better than Windows )
*Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-208-generic x86_64)*
*root@Discourse:~# cd /var/discourse/*
*root@Discourse:/var/discourse# ./launcher restart app*
*ERROR: Docker version 19.03.13 not supported, please upgrade to at least 20.10.0, or recommended 24.0.7*
*root@Discourse:/var/discourse# sudo apt-get install docker-ce --only-upgrade*
*Reading package lists... Done*
*Building dependency tree*
*Reading state information... Done*
*docker-ce is already the newest version (5:19.03.13~3-0~ubuntu-xenial).*
******(/(&(%%(&/()(=)/&/))&/*
******Didn't wanna toss docker and reinstall it but now I've had enough...*
*root@Discourse:/var/discourse# sudo apt remove docker docker-engine docker.io containerd runc*
*root@Discourse:/var/discourse# sudo apt install apt-transport-https ca-certificates curl software-properties-common*
*root@Discourse:/var/discourse# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -*
*root@Discourse:/var/discourse# sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"*
*root@Discourse:/var/discourse# sudo apt update*
******Was going to run this as well but couldn't find it in my logs so maybe I didn't (command doesn't show but install does, huh?):*
******sudo apt install docker-ce docker-ce-cli containerd.io*
*root@Discourse:/var/discourse# sudo docker --version*
*Docker version 28.0.1, build 068a01e*
*root@Discourse:/var/discourse# ./launcher rebuild app*
****** Took a few tries but finally looked better after a few rebuilds.*
******... finally got to redis, woohooo... or not*
*docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint app (ea1a6cc790c13940435c2626a4e8b6169f04612f4be552be25564def7d5745eb): failed to bind host port for 0.0.0.0:80:172.17.0.2:80/tcp: address already in use*
******(/(&(%%(&/()(=)/&/))&/ (or did I say that already?)*
*root@Discourse:/var/discourse# hostname -I*
*139.X.X.X 10.19.0.5 172.17.0.1*
*root@Discourse:/var/discourse# sudo lsof -i :80*
*COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME*
*apache2 839 root 4u IPv6 23477 0t0 TCP *:http (LISTEN)*
*apache2 844 www-data 4u IPv6 23477 0t0 TCP *:http (LISTEN)*
*apache2 845 www-data 4u IPv6 23477 0t0 TCP *:http (LISTEN)*
******Apache on by default? Since when? Hell no!*
*root@Discourse:/var/discourse# sudo systemctl stop apache2*
*root@Discourse:/var/discourse# sudo systemctl disable apache2*
*Synchronizing state of apache2.service with SysV service script with /lib/systemd/systemd-sysv-install.*
*Executing: /lib/systemd/systemd-sysv-install disable apache2*
*Removed /etc/systemd/system/multi-user.target.wants/apache2.service.*
*root@Discourse:/var/discourse# ./launcher start app*
******Hey presto, finally! (After the usual slow ngix puke message)*
******After booting I of course get... which I was waiting for... oh well, once everything works for a week, maybe continue suffering... after taking a snapshot and backup and... ;) *
*New release '22.04.5 LTS' available.*
*Run 'do-release-upgrade' to upgrade to it.*
I would have created a new droplet and either just done a fresh install and copied over the backup or Move a Discourse site to another VPS with rsync. I also do a hybrid version where I rsync over the ssl certs, but then restore a backup. This saves the step of upgrading the database.
.
.
.
)