Discourse image and installation size. Clean /var/lib/docker/overlay2?

[m4rv1n]

Hello, I installed discourse on a new and dedicated machine following the guide at https://github.com/discourse/discourse/blob/main/docs/INSTALL-cloud.md .

The server has 15GB of disk and also if the installation is not prduction running (less than 10 users, no attachments, some post created and deleted) it seems the installation size is rather big.

#df -kh
Filesystem                         Size  Used Avail Use% Mounted on
udev                               950M     0  950M   0% /dev
tmpfs                              199M  1.3M  198M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   15G   11G  3.2G  78% /
tmpfs                              994M     0  994M   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              994M     0  994M   0% /sys/fs/cgroup
/dev/sda2                          976M  203M  707M  23% /boot
/dev/loop0                          56M   56M     0 100% /snap/core18/2066
/dev/loop1                          56M   56M     0 100% /snap/core18/2074
/dev/loop2                          33M   33M     0 100% /snap/snapd/12398
/dev/loop3                          33M   33M     0 100% /snap/snapd/12159
/dev/loop4                          68M   68M     0 100% /snap/lxd/20326
overlay                             15G   11G  3.2G  78% /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/merged
/dev/loop6                          71M   71M     0 100% /snap/lxd/21029
tmpfs                              199M     0  199M   0% /run/user/1000

# du -csh /var/lib/docker/overlay2
580M    51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****
2.3G    ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****
76M     d1b8d94d2ecfa140794c61e2a81ad4a09eba1646d764018cf5afd433b51*****
4.5G    ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****
40K     ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****-init
24K     l
7.4G    total

# du -shc /var/lib/docker/overlay2/*/diff
580M    /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****/diff
2.3G    /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff
76M     /var/lib/docker/overlay2/d1b8d94d2ecfa140794c61e2a81ad4a09eba1646d764018cf5afd433b51*****/diff
996M    /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/diff
20K     /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****-init/diff
3.9G    total

# docker images -a
REPOSITORY            TAG       IMAGE ID       CREATED        SIZE
local_discourse/app   latest    b29b7073fea2   2 months ago   2.69GB
<none>                <none>    30e4746e631e   3 months ago   2.23GB

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         2.689GB   0B (0%)
Containers      1         1         950.4MB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Tried this but didn’t help

# /var/discourse/launcher cleanup
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

I am worried once in production the installation will eat more and more space and it could break.
Could you advice what is the normal size of the docker image and installation and what can I do to free up some space?
Thanks

[Ed_S]

I’ll be interested to know what’s the meaning of the <none> docker image. (Edit: it seems, if a <none> image shows up in docker images -a that’s not too important, but if it shows up in docker images then it is a waste of space. I would hope launcher cleanup would help, but it hasn’t helped you…)

Note that the overlay filesystem usage as seen by df matches the usage of the root filesystem: there’s a lot of data doing double duty and you need to take care not to double-count. In your case the available space is 3.2G and that’s the figure to worry about. There might well be some tidying up to do.

I’ll show my similar stats below. I have two forums, each on a different host. One host has 20G and the other has 25G of space. I think 15G might prove to be very tight, especially when the update process demands 5G free space before starting.

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   19G  5.1G  79% /

# docker images -a
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app   latest              8da0107aba03        2 months ago        2.7GB
discourse/base        2.0.20210415-1332   30e4746e631e        3 months ago        2.23GB
<none>                <none>              1e6bf44c2762        5 months ago        2.46GB
discourse/base        2.0.20201221-2020   c0704d4ce2b4        7 months ago        2.11GB


# du -shc /var/lib/docker/overlay2/*/diff
2.2G	/var/lib/docker/overlay2/05fa0e4df2...
76M 	/var/lib/docker/overlay2/58b000b1f5c...
20K  	/var/lib/docker/overlay2/6271023fc7a...
1.1G	/var/lib/docker/overlay2/6271023fc7...
2.3G	/var/lib/docker/overlay2/91d6adf7ad...
481M	/var/lib/docker/overlay2/b6b06a7cee...
592M	/var/lib/docker/overlay2/d81e44d563...
76M 	/var/lib/docker/overlay2/fb98649680b...
6.8G	total


# du -shc /var/lib/docker/overlay2/*
2.2G	/var/lib/docker/overlay2/05fa0e4df2...
76M 	/var/lib/docker/overlay2/58b000b1f5c...
4.7G	/var/lib/docker/overlay2/6271023fc7...
40K  	/var/lib/docker/overlay2/6271023fc7a...
2.3G	/var/lib/docker/overlay2/91d6adf7ad...
481M	/var/lib/docker/overlay2/b6b06a7cee...
592M	/var/lib/docker/overlay2/d81e44d563...
76M 	/var/lib/docker/overlay2/fb98649680b...
36K  	/var/lib/docker/overlay2/l
11G	total


# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              4                   1                   5.155GB             4.689GB (90%)
Containers          1                   1                   1.059GB             0B (0%)
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

See perhaps also Fixing discourse after disk full :

@Ed_S

In another case, I use Digital Ocean’s lowest tier, offering 25G, and in that case the forum’s backup comes in at 640MB.)

and Minimum requirements to use Discourse? :

@Ed_S

By way of calibration, I have two small forums:

and 2.6.0 beta 3 update failed on disk and/or memory space:

@Ed_S

You can try to find out, for example at your root shell prompt:
du -kx / | sort -n | tail -33

[Ed_S]

An update: as I’ve taken a backup and run an update, I also ran a cleanup.

I’m fairly sure that the tools to manage docker images are preferable to direct action on /var/lib/docker/overlay2. And the automatic cleanup is preferable to those tools. (But as noted in the linked posts, there are other ways in which disk space gets used: backups, swapfiles, journal files, uploads, imports, log files, and so on)

Here’s what I saw, as an ‘after’ to the ‘before’ posted above - note that the starting point is lower on free disk space, probably because of the backup:

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   19G  5.1G  79% /
# docker images -a
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app   latest              8da0107aba03        2 months ago        2.7GB
discourse/base        2.0.20210415-1332   30e4746e631e        3 months ago        2.23GB
<none>                <none>              1e6bf44c2762        5 months ago        2.46GB
discourse/base        2.0.20201221-2020   c0704d4ce2b4        7 months ago        2.11GB
# /var/discourse/launcher cleanup
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: discourse/base:2.0.20201221-2020
untagged: discourse/base@sha256:e18*
deleted: sha256:1e6*
deleted: sha256:a22*
deleted: sha256:c07*
deleted: sha256:9b7*
deleted: sha256:87c*
untagged: discourse/base:2.0.20210415-1332
untagged: discourse/base@sha256:b3b*

Total reclaimed space: 2.456GB
# docker images -a
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app   latest              8da0107aba03        2 months ago        2.7GB
<none>                <none>              30e4746e631e        3 months ago        2.23GB
# docker images 
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
local_discourse/app   latest              8da0107aba03        2 months ago        2.7GB
# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   17G  7.8G  68% /
# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1                   1                   2.699GB             0B (0%)
Containers          1                   1                   1.13GB              0B (0%)
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B
# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   17G  7.8G  68% /
# du -shc /var/lib/docker/overlay2/*
4.9G	/var/lib/docker/overlay2/627*
40K 	/var/lib/docker/overlay2/627*-init
2.3G	/var/lib/docker/overlay2/91d*
592M	/var/lib/docker/overlay2/d81*
76M 	/var/lib/docker/overlay2/fb9*
24K 	/var/lib/docker/overlay2/l
7.8G	total
# du -shc /var/lib/docker/overlay2/*/diff
20K 	/var/lib/docker/overlay2/627*-init/diff
1.2G	/var/lib/docker/overlay2/627*/diff
2.3G	/var/lib/docker/overlay2/91d*/diff
592M	/var/lib/docker/overlay2/d81*/diff
76M 	/var/lib/docker/overlay2/fb9*/diff
4.2G	total

[pfaffman (Jay Pfaffman)]

I’d say that you need at least 25gb, but @Ed_S said he got it to work with 20. And even 25 is a bit tight in my experience.

[m4rv1n]

@Ed_S

@Ed_S

du -kx / | sort -n | tail -33

Hi all,
thanks for your reply.

I checked the journal with

# journalctl --disk-usage
Archived and active journals take up 1.5G in the file system.

and run this command to free the space
# journalctl --vacuum-time=10d

Now I have 1GB more.
My pure question was about the real size needed from discourse.
Actually my backups, because not in production, is around 10MB, for a total of 3 backups (30MB).

I understand the need of space during the update, but during running I expect to save current version and have the chance to delete old versions.
Do we need all of the diff in the overlay or can we merge all the content in some way to recover space? I also don’t see any mounted volume so imagine discourse save there all the data.

@Ed_S here the result of the command

~# du -kx / | sort -n | tail -33
431172  /usr/bin
482000  /var/lib/snapd
499660  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse/vendor/bundle/ruby/2.7.0/gems
533056  /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****/diff/var/www/discourse
533060  /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****/diff/var/www
556800  /usr/lib/firmware
570876  /usr/lib/modules
574032  /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****/diff/var
593840  /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****/diff
593856  /var/lib/docker/overlay2/51d029d96e73b67e449f0b0570be47b6292da46c8c69b9f0dc6df35db85*****
626400  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse/vendor/bundle/ruby/2.7.0
626404  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse/vendor/bundle/ruby
626408  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse/vendor/bundle
634964  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse/vendor
845496  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/usr/lib
863600  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www/discourse
863612  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var/www
936876  /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/var
1004276 /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/diff/var/www/discourse
1004284 /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/diff/var/www
1032452 /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/diff/var
1091584 /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****/diff
1091604 /var/lib/docker/overlay2/ef92e2dc7a656c20eccbbdd40e660c76631ef48b6989f9ded3889a929eb*****
1426272 /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff/usr
1579980 /usr/lib
2398720 /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****/diff
2398736 /var/lib/docker/overlay2/ae38f397c79178185e26b87a9e2c6c890b0db9d3456de6d34ae3c3b96db*****
2607528 /usr
4161292 /var/lib/docker/overlay2
4171308 /var/lib/docker
4816636 /var/lib
5509688 /var
10220684        /

[Ed_S]

In the linked thread we see a situation where ./launcher cleanup didn’t help but docker system prune --all --volumes --force did help. I don’t know why that would be.

I don’t understand the way the docker images stackup, why the diff directories are so large, or why the docker disk usage comes out much bigger than docker’s report of sizes.

However, it’s worth noting that minimising disk usage takes effort: neither docker nor discourse make a priority of minimising disk usage. Both projects prefer to work on improving the product and keeping things simple.

I am successfully running on a 20G instance. Given the need to keep 5G free for upgrades, that’s actually a limit of 15G on used disk space. In your case, with a 15G instance, you’d be trying to stay within 10G used. We can expect that to be more difficult, maybe even not possible. There’s surely some minimum.

I’m pretty sure I saw a thread where the advice was to remove all of docker and discourse, leaving just the site configuration and the database, and reinstall. I wouldn’t do that lightly, and I certainly wouldn’t do it without a backup! Unfortunately I can’t find that thread presently.

[m4rv1n]

My “docker volume ls” is empty , it depends if people mounted volume.

It would be nice to have some more official specification or expected size of the installation, maybe there are somewhere but I didn’t find.
I understand in today world 20GB is not that much but when it comes to backup, remote transfer, duplication and so on it is a waste of space that maybe can be avoided if we know how to save this space at the beginning.

The non-docker installation is possible but as far as I know it is not supported if any problem.

[pfaffman (Jay Pfaffman)]

@m4rv1n

I understand in today world 20GB is not that much

And 25gb is what the typical novice and low budget user who uses digital ocean gets. That’s really the target audience.

@m4rv1n

if we know how to save this space at the beginning.

The set of people who have the expertise to manage a smaller than 25gb disk space who don’t have the budget for 25gb of disk space is very small. I think you’re the first person who has had your request in the past 5 years.

@m4rv1n

maybe can be avoided if we know how to save this space at the beginning.

You could save a bit by managing your own backups and backing up remotely. A single backup requires three copies. The old copy, the new uncompressed copy, and the new compressed copy. That’s the biggest space saver I can think of, but I don’t claim to have lots of special docker knowledge.

You could also contrive to build the new image remotely on a temporary server and push it to a repo. Then you’d not need the extra docker space locally. That is probably your best bet, but there’s no well-documented way to do that and it is mostly not supported (but more so than a non docker install). If you were to hire me to do it, it would cost years of what 10gb of disk space costs. I’ve considered a service where I’d build those images (essentially a better supported bitnami) but haven’t gotten around to it because I don’t think there’s a market to justify the development time.

[Ed_S]

I’d be interested to know how much your 15G server costs, and from which supplier. My 25G machine costs me $6 per month from Digital Ocean, and my 20G machine costs $3 from Hetzner.

If you don’t expect to have to debug any behaviour from system logs, you can be very brutal in pruning the journals.

# journalctl --rotate
# journalctl --vacuum-time=1s

although what I’ve done is change the policy:

/etc/systemd/journald.conf:
[Journal]
SystemMaxUse=50M

The disk usage on the 20G machine, which has a very small low-activity forum on it, is like this:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        19G  9.9G  8.1G  56% /

operating system:

15M /bin
16M /sbin
43M /boot/
135M /var/cache/
223M /var/log
435M /lib
464M /var/lib/apt
1.1G /usr
2.1G /swapfile
4.3G total

discourse:

# du -hs /var/discourse/ /var/lib/docker/
1.5G /var/discourse/
7.5G /var/lib/docker/

edit: aha, but we’re double-counting, see instead

# du -hxs /var/discourse/ /var/lib/docker/
1.5G /var/discourse/
4.1G /var/lib/docker/

the forum itself, within the above:

201M	/var/discourse/shared/standalone/uploads
315M	/var/discourse/shared/standalone/postgres_data
930M	/var/discourse/shared/standalone/backups

I’m running Ubuntu 18.04 whereas it looks like you’re running something more like 20.04 which is probably bigger.

[m4rv1n]

I use different VPS, I have some VM on contabo, others on Azure, for me the question to know if what I maintain is maintained and configured as expected (size, cleaning, etc).

Thanks for the tip on changing policy for journald, I did it (I cleand before so just little saving, but at least now I don’t have to check it).

Your disk usage is more or less equal to mine

# du -hxs /var/discourse/ /var/lib/docker/
181M    /var/discourse/
4.0G    /var/lib/docker/

I would go for a deletion of docker and rebuild, but because no fixed volume is mounted for docker I am worried to lose forum data at the deletion stage. I will wait for more official procedure on it.

[Ed_S]

@m4rv1n

I would go for a deletion of docker and rebuild, but because no fixed volume is mounted for docker I am worried to lose forum data at the deletion stage. I will wait for more official procedure on it.

Yes, worth looking for confirmation. I’m pretty sure all the forum data is under /var/discourse.

Another approach which might be available to you is to spin up a new instance and install discourse fresh, and see how it looks. (One should in any case test one’s backups!)