2.6.0 beta 3 update failed on disk and/or memory space

I wonder, if in addition to disk space issues, the upgrade process exceeded available memory (1GB) allocated to the droplet? You can see in my console screenshot above a reference to ‘Out of memory’ as the first item logged after a ./launcher rebuild app

What I did not mention, is that after that attempt, console stopped responding (though at that point I was using the web based console in my Digital Ocean Control Panel, which is always flaky) and I then power cycled the droplet. (Thereafter I used PuTTY)

Either way, no, not great that update was reported as a success via Discourse upgrade page after presumably hitting the same memory and/or disk issue.


Ah, the OOM killer ran. That’s certainly not good. Normally I would recommend increasing swap space. You can see current usage with swapon, in my case

# swapon 
/swapfile file   2G   3M   -2

Also free

# free
              total        used        free      shared  buff/cache   available
Mem:        1992060      792904       80148       34696     1119008     1004956
Swap:       2097148        3084     2094064

It would be bad if your 2G swapfile was not in play. It’s bad that you can’t add swap without using disk space!

One way to improve disk space for an upgrade is to copy all the backup files offsite, check their integrity, then delete them from your server. You absolutely need a good recent backup somewhere safe over an upgrade, just in case, but it need not be on the server itself. I would feel comfortable deleting all but the latest, but would certainly take an offline copy.

It would be good to see the du results again, now you’ve done all the cleanups.


I wonder: is the 1G your RAM allocation and the 25G your disk allocation? Two very different things.

Edit: the supported standard story is, I think, to have rather more than 1G of RAM.
Edit: no, apparently 1G is still the recommended absolute minimum.


I just connected again , and system info reported on launch of console window is

System load:  0.01               Processes:              136
Usage of /:   59.4% of 24.06GB   Users logged in:        0
Memory usage: 73%                IP address for eth0:
Swap usage:   17%                IP address for docker0:

So swap space of 17% = 4GB?
With no one logged in to the forum, and only the current PuTTY connection to droplet active, RAM is 73% full - so it doesn’t look like it would take much activity to tip the forum over into swap space - and if that comes out of the 24GB then perhaps that creates the perfect storm during an update with disk space usage already running high?

du -kx / | sort -n | tail -33 now gives me

root@nz:~# du -kx / | sort -n | tail -33
505512  /usr/bin
528784  /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle/ruby/2.6.0
528788  /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle/ruby
528792  /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle
536848  /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor
548952  /var/lib/docker/overlay2/c126267f944d8d7f12415ac4f5908eba8a6a686b093cad3e0115eded8edfd6ba/diff
548968  /var/lib/docker/overlay2/c126267f944d8d7f12415ac4f5908eba8a6a686b093cad3e0115eded8edfd6ba
817700  /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/usr/lib
827812  /var/log/journal/8bebc832e1a692c83690ffe65e1256e3
868792  /var/log/journal
1069356 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse
1069368 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www
1069396 /var/log
1142352 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var
1202004 /var/discourse/shared/standalone/import/data
1307816 /var/discourse/shared/standalone/import
1362804 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/usr
1399332 /var/discourse/shared/standalone/backups/default
1399336 /var/discourse/shared/standalone/backups
1709408 /usr
2438224 /var/discourse/shared/standalone/postgres_data/base/16583
2462944 /var/discourse/shared/standalone/postgres_data/base
2481288 /var/discourse/shared/standalone/postgres_data
2540188 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff
2540204 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35
3387776 /var/lib/docker/overlay2
3460136 /var/lib/docker
3830584 /var/lib
5629420 /var/discourse/shared/standalone
5629504 /var/discourse/shared
5632224 /var/discourse
10747244        /var
14961492        /
1 Like

I think you can improve this with journalctl maybe with

# journalctl --vacuum-size=50M

(which you might do immediately before trying an upgrade)

Interesting that the postgresql usage hasn’t gone down.

free will show you the swap usage: it’s 17% used, of some amount, probably 2G.

It’s clear that your machine is a little uncomfortably small: you need more RAM or more swap, and you can’t practically have much more swap without getting more disk.

1 Like

Apologies - you are quite right. the 1GB was RAM, not disk space used.

1 Like

Quite right again

root@nz:~# free
              total        used        free      shared  buff/cache   available
Mem:        1008828      655660       61716      102288      291452       96576
Swap:       2097148      459776     1637372

I wonder if the upgrade process should assess the host system for capacity to implement the upgrade just before it starts?


I think it’s somewhat in the category of predicting the future! The check for 5G disk space is clearly helpful, but won’t be watertight. Free RAM is more difficult, it’s quite slippery as to how much will be needed. It will be a function, I would think, of how big the forum has become, and perhaps also of what needs to be touched during each upgrade.

I’m careful to minimise costs, so I will spend time trying to squeeze into a cheap server. But eventually, as a forum grows, it will surely be worth moving up to the next tier. And that will cost money, but save time and effort.


I am very much at the ‘minimize costs’ end of the scale sadly - the forum is a purely volunteer based effort generating no revenue, and with the addition of email based posting (via MailGun) demanded by users, it can cost me a bit each month to keep running as it is.

As hobbies go, cheaper than drinking in nightclubs I guess!


This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.