I wonder, if in addition to disk space issues, the upgrade process exceeded available memory (1GB) allocated to the droplet? You can see in my console screenshot above a reference to ‘Out of memory’ as the first item logged after a ./launcher rebuild app
What I did not mention, is that after that attempt, console stopped responding (though at that point I was using the web based console in my Digital Ocean Control Panel, which is always flaky) and I then power cycled the droplet. (Thereafter I used PuTTY)
Either way, no, not great that update was reported as a success via Discourse upgrade page after presumably hitting the same memory and/or disk issue.
Ah, the OOM killer ran. That’s certainly not good. Normally I would recommend increasing swap space. You can see current usage with swapon, in my case
# swapon
NAME TYPE SIZE USED PRIO
/swapfile file 2G 3M -2
Also free
# free
total used free shared buff/cache available
Mem: 1992060 792904 80148 34696 1119008 1004956
Swap: 2097148 3084 2094064
It would be bad if your 2G swapfile was not in play. It’s bad that you can’t add swap without using disk space!
One way to improve disk space for an upgrade is to copy all the backup files offsite, check their integrity, then delete them from your server. You absolutely need a good recent backup somewhere safe over an upgrade, just in case, but it need not be on the server itself. I would feel comfortable deleting all but the latest, but would certainly take an offline copy.
It would be good to see the du results again, now you’ve done all the cleanups.
I just connected again , and system info reported on launch of console window is
System load: 0.01 Processes: 136
Usage of /: 59.4% of 24.06GB Users logged in: 0
Memory usage: 73% IP address for eth0: 159.65.140.176
Swap usage: 17% IP address for docker0: 172.17.0.1
So swap space of 17% = 4GB?
With no one logged in to the forum, and only the current PuTTY connection to droplet active, RAM is 73% full - so it doesn’t look like it would take much activity to tip the forum over into swap space - and if that comes out of the 24GB then perhaps that creates the perfect storm during an update with disk space usage already running high?
I think you can improve this with journalctl maybe with
# journalctl --vacuum-size=50M
(which you might do immediately before trying an upgrade)
Interesting that the postgresql usage hasn’t gone down.
free will show you the swap usage: it’s 17% used, of some amount, probably 2G.
It’s clear that your machine is a little uncomfortably small: you need more RAM or more swap, and you can’t practically have much more swap without getting more disk.
I think it’s somewhat in the category of predicting the future! The check for 5G disk space is clearly helpful, but won’t be watertight. Free RAM is more difficult, it’s quite slippery as to how much will be needed. It will be a function, I would think, of how big the forum has become, and perhaps also of what needs to be touched during each upgrade.
I’m careful to minimise costs, so I will spend time trying to squeeze into a cheap server. But eventually, as a forum grows, it will surely be worth moving up to the next tier. And that will cost money, but save time and effort.
I am very much at the ‘minimize costs’ end of the scale sadly - the forum is a purely volunteer based effort generating no revenue, and with the addition of email based posting (via MailGun) demanded by users, it can cost me a bit each month to keep running as it is.
As hobbies go, cheaper than drinking in nightclubs I guess!