You can try to find out, for example at your root shell prompt:
du -kx / | sort -n | tail -33
and perhaps also
find / -xdev -size +1000k -ls | sort -n -k 2 | tail
3 Likes
OK, du command (=disk usage?) reports
root@nz:/var/discourse# du -kx / | sort -n | tail -33
747432 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff/var/www/discourse/vendor/bundle
750792 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff/var/www/discourse/vendor
819620 /var/log/journal/8bebc832e1a692c83690ffe65e1256e3
860600 /var/log/journal
926492 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff/var/www/discourse
926504 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff/var/www
998192 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff/var
1059968 /var/log
1202004 /var/discourse/shared/standalone/import/data
1237720 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff/usr
1307816 /var/discourse/shared/standalone/import
1399332 /var/discourse/shared/standalone/backups/default
1399336 /var/discourse/shared/standalone/backups
1636032 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff/var/www/discourse
1636040 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff/var/www
1678944 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff/var
1692708 /usr
1820540 /var/lib/docker/volumes
2235084 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5/diff
2235104 /var/lib/docker/overlay2/794d92242be8354a3ba468161d1d2ea416ab43a6f70ddd7ab060384bd5a9fcd5
2285628 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff
2285644 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354
2437984 /var/discourse/shared/standalone/postgres_data/base/16384
2461228 /var/discourse/shared/standalone/postgres_data/base
2545304 /var/discourse/shared/standalone/postgres_data
5184184 /var/lib/docker/overlay2
5692972 /var/discourse/shared/standalone
5693056 /var/discourse/shared
5695776 /var/discourse
7076692 /var/lib/docker
7446508 /var/lib
14417296 /var
18614840 /
root@nz:/var/discourse#
18.6 GB?
The find command gives
root@nz:/var/discourse# find / -xdev -si ze +1000k -ls | sort -n -k 2 | tail
524219 217044 -rw------- 1 lxd mlocate 222248960 May 22 03:39 /var/discourse/shared/standalone/postgres_data/base/16384/20379
523733 234132 -rw------- 1 lxd mlocate 239747072 May 23 01:29 /var/discourse/shared/standalone/postgres_data/base/16384/19615
610933 244612 -rw------- 1 lxd mlocate 250478592 Sep 25 03:37 /var/discourse/shared/standalone/postgres_data/base/16384/153088
610816 248180 -rw------- 1 lxd mlocate 254132224 Sep 26 03:59 /var/discourse/shared/standalone/postgres_data/base/16384/152767
2596079 307188 -r--r--r-- 1 ReadyNAS ReadyNAS 314556297 Apr 29 20:50 /var/lib/docker/overlay2/fadc3460a6249ba37efa266d885f5ea3d6f6de8fee0e9b056c5376599bb7f354/diff/var/www/discourse/.git/objects/pack/pack-ffd1b8da21b9e26b4475a3fef6537a89f21989d6.pack
1548730 464076 -rw-r--r-- 1 ReadyNAS www-data 475209681 Sep 24 03:31 /var/discourse/shared/standalone/backups/default/nz-architecture-2020-09-24-033014-v20200820232017.tar.gz
1548729 467616 -rw-r--r-- 1 ReadyNAS www-data 478832984 Sep 25 03:38 /var/discourse/shared/standalone/backups/default/nz-architecture-2020-09-25-033723-v20200820232017.tar.gz
1548294 467636 -rw-r--r-- 1 ReadyNAS www-data 478853932 Sep 26 03:54 /var/discourse/shared/standalone/backups/default/nz-architecture-2020-09-26-035209-v20200820232017.tar.gz
794198 627936 -rw-r--r-- 1 ReadyNAS ReadyNAS 643002368 Mar 28 2020 /var/discourse/shared/standalone/import/data/index.db
60454 2097156 -rw------- 1 root root 2147483648 Jan 4 2020 /swapfile
root@nz:/var/discourse#
Does anything stand out as unexpectedly large ?
I have no way to tell what is normal, and what if anything can be safely deleted…
(the ReadyNAS files are forum backups, generated by Discourse)
1 Like
Database is 2.5GB. Backups combined are 1.3GB.
You’ve got several large Docker layers - are you sure that ./launcher cleanup actually completed?
3 Likes
./launcher cleanup gives following result for me, almost instantaneously - not sure what I am missing here…
root@nz:/var/discourse# ./launcher cleanup
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
root@nz:/var/discourse#
1 Like
That large import area might be suspicious. But it’s only 640M.
There are some tips here about seeing what docker is using, and trying to reduce it. (It’s possible some of the advice is wrong or dangerous.) I saw this in my rather small world:
# docker volume ls -qf dangling=true
# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
local_discourse/app latest 33ce56b34841 3 months ago 2.59GB
<none> <none> 991acdba0b1f 4 months ago 2.22GB
# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 1 2.591GB 0B (0%)
Containers 1 1 920.9MB 0B (0%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
2 Likes
Did you see this advice about vacuuming your database and rebuilding indexes after the recent postgresql update?
4 Likes
Thanks Ed - both those things helped
Most posts in that thread did not help, but one suggested docker system prune --all --volumes --force which cleaned more than 1GB
That procedure reclaimed 2.3GB - enough to give me the minimum 5GB
I then repeated the first two commands of
But the process seemed to fill things back up again, because at the third command I got
root@nz:/var/discourse# ./launcher rebuild app
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 25G 22G 3.0G 88% /
Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)y
If the cleanup was successful, you may try again now
At that point I tried ./launcher cleanup again, but got
root@nz:/var/discourse# /launcher cleanup
-bash: /launcher: No such file or directory
2 Likes
(That should be ./launcher not /launcher)
2 Likes
Ah - thanks Ed! That got back 2.6GB - just enough to let installation complete.
Many thanks to everyone for the helpful suggestions.
It still worries me a bit that I am apparently using so much space, at least as reported with Ubuntu - because future forum updates may presumably hit the same issue.
I note within my Discourse installation, free disk space available for storing backups is reported as 9 GB - a major discrepancy. What could lie behind that?
** Edit ** - retrying du -kx / | sort -n | tail -33 I see Discourse is correct - my disk usage has shrunk to 15GB as a result of all of the above
1 Like
if you go to yourdomain.com/admin/backups
How many backups do you have there? maybe you can keep one and delete the rest.
2 Likes
Only 3 backups now @ 457 MB each - but looks like disk space headroom problem has resolved through the various clean ups… for now at least.
I read in passing something about dockers accumulating ‘difference data’ and logs at a ferocious rate, and a lot of people reporting similar issues with disks filling themselves up
1 Like
I would agree that this following message is worrying, and it’s odd that discourse believes it is fully updated:
2 Likes
I wonder, if in addition to disk space issues, the upgrade process exceeded available memory (1GB) allocated to the droplet? You can see in my console screenshot above a reference to ‘Out of memory’ as the first item logged after a ./launcher rebuild app
What I did not mention, is that after that attempt, console stopped responding (though at that point I was using the web based console in my Digital Ocean Control Panel, which is always flaky) and I then power cycled the droplet. (Thereafter I used PuTTY)
Either way, no, not great that update was reported as a success via Discourse upgrade page after presumably hitting the same memory and/or disk issue.
2 Likes
Ah, the OOM killer ran. That’s certainly not good. Normally I would recommend increasing swap space. You can see current usage with swapon, in my case
# swapon
NAME TYPE SIZE USED PRIO
/swapfile file 2G 3M -2
Also free
# free
total used free shared buff/cache available
Mem: 1992060 792904 80148 34696 1119008 1004956
Swap: 2097148 3084 2094064
It would be bad if your 2G swapfile was not in play. It’s bad that you can’t add swap without using disk space!
One way to improve disk space for an upgrade is to copy all the backup files offsite, check their integrity, then delete them from your server. You absolutely need a good recent backup somewhere safe over an upgrade, just in case, but it need not be on the server itself. I would feel comfortable deleting all but the latest, but would certainly take an offline copy.
It would be good to see the du results again, now you’ve done all the cleanups.
2 Likes
I wonder: is the 1G your RAM allocation and the 25G your disk allocation? Two very different things.
Edit: the supported standard story is, I think, to have rather more than 1G of RAM.
Edit: no, apparently 1G is still the recommended absolute minimum.
2 Likes
I just connected again , and system info reported on launch of console window is
System load: 0.01 Processes: 136
Usage of /: 59.4% of 24.06GB Users logged in: 0
Memory usage: 73% IP address for eth0: 159.65.140.176
Swap usage: 17% IP address for docker0: 172.17.0.1
So swap space of 17% = 4GB?
With no one logged in to the forum, and only the current PuTTY connection to droplet active, RAM is 73% full - so it doesn’t look like it would take much activity to tip the forum over into swap space - and if that comes out of the 24GB then perhaps that creates the perfect storm during an update with disk space usage already running high?
du -kx / | sort -n | tail -33 now gives me
root@nz:~# du -kx / | sort -n | tail -33
505512 /usr/bin
528784 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle/ruby/2.6.0
528788 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle/ruby
528792 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor/bundle
536848 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse/vendor
548952 /var/lib/docker/overlay2/c126267f944d8d7f12415ac4f5908eba8a6a686b093cad3e0115eded8edfd6ba/diff
548968 /var/lib/docker/overlay2/c126267f944d8d7f12415ac4f5908eba8a6a686b093cad3e0115eded8edfd6ba
817700 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/usr/lib
827812 /var/log/journal/8bebc832e1a692c83690ffe65e1256e3
868792 /var/log/journal
1069356 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www/discourse
1069368 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var/www
1069396 /var/log
1142352 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/var
1202004 /var/discourse/shared/standalone/import/data
1307816 /var/discourse/shared/standalone/import
1362804 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff/usr
1399332 /var/discourse/shared/standalone/backups/default
1399336 /var/discourse/shared/standalone/backups
1709408 /usr
2438224 /var/discourse/shared/standalone/postgres_data/base/16583
2462944 /var/discourse/shared/standalone/postgres_data/base
2481288 /var/discourse/shared/standalone/postgres_data
2540188 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35/diff
2540204 /var/lib/docker/overlay2/3b68a713bd8e9a7f3b2a69ba8084a770b796e555e887ce4f66698d3894430c35
3387776 /var/lib/docker/overlay2
3460136 /var/lib/docker
3830584 /var/lib
5629420 /var/discourse/shared/standalone
5629504 /var/discourse/shared
5632224 /var/discourse
10747244 /var
14961492 /
root@nz:~#
1 Like
I think you can improve this with journalctl maybe with
# journalctl --vacuum-size=50M
(which you might do immediately before trying an upgrade)
Interesting that the postgresql usage hasn’t gone down.
free will show you the swap usage: it’s 17% used, of some amount, probably 2G.
It’s clear that your machine is a little uncomfortably small: you need more RAM or more swap, and you can’t practically have much more swap without getting more disk.
1 Like
Apologies - you are quite right. the 1GB was RAM, not disk space used.
1 Like
Quite right again
root@nz:~# free
total used free shared buff/cache available
Mem: 1008828 655660 61716 102288 291452 96576
Swap: 2097148 459776 1637372
I wonder if the upgrade process should assess the host system for capacity to implement the upgrade just before it starts?
2 Likes
I think it’s somewhat in the category of predicting the future! The check for 5G disk space is clearly helpful, but won’t be watertight. Free RAM is more difficult, it’s quite slippery as to how much will be needed. It will be a function, I would think, of how big the forum has become, and perhaps also of what needs to be touched during each upgrade.
I’m careful to minimise costs, so I will spend time trying to squeeze into a cheap server. But eventually, as a forum grows, it will surely be worth moving up to the next tier. And that will cost money, but save time and effort.
2 Likes