Docker uses root partition, might need more storage space

[Ironclad14]

Hi guys,

First post here, sadly it’s regarding support. Docker seems to use my /root partition which is only 20 GB and the rest of my 2TB drive is on /home.

root@legio:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        20G  8.3G  9.9G  46% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G  4.0K  7.8G   1% /dev/shm
tmpfs           7.8G  394M  7.4G   5% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/md2        487M   25M  433M   6% /boot
/dev/md4        1.8T   51G  1.7T   3% /home
overlay          20G  8.3G  9.9G  46% /var/lib/docker/overlay2/aa30e247cdd1f95098ce909f333aae8a968b431c07856f242614528d6e2490c3/merged
shm              64M  4.0K   64M   1% /var/lib/docker/containers/201cd5c089c15b57b38fc94b3331212494d0a9514ea6ca51ec88bbcacce0cf0a/shm
overlay          20G  8.3G  9.9G  46% /var/lib/docker/overlay2/d249b6017bcfdae9742caed4402fe02936b1ddb55eb5178504d161061d49f178/merged
shm              64M     0   64M   0% /var/lib/docker/containers/0b11fcac390be8525e451bf00270916063c7da519c368e170c49f5f741c1b425/shm

Can you let me know of any way I can make use of more space without having to resize partitions? It’s risky business while they’re mounted and data loss isn’t something anyone likes to deal with.

Running Debian 9.1 stable on a dedicated machine.

Any help will be appreciated.

[codinghorror (Jeff Atwood)]

Do we have a howto for this @techapj? That is, configuring Discourse to use another drive for the high storage requirement items like the database, file/image uploads, backups, etc?

[Ironclad14]

That would be the easiest solution, yeah.

[dax (Daniela)]

This topic, though related to digital ocean block storage, can help you at least for uploads:

Need a how-to guide for using DigitalOcean Block Storage for image uploads

The man makes a point. It appears to be less than 1/3 the speed on the simplest of benchmarks. (I checked and both the host and the storage are at SFO2.) root#:dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k 1024+0 records in 1024+0 records out 402653184 bytes (403 MB, 384 MiB) copied, 0.590498 s, 682 MB/s root#:dd if=/dev/zero of=/mnt/volume-sfo2-01/test conv=fdatasync bs=384k count=1k 1024+0 records in 1024+0 records out 402653184 bytes (403 MB, 384 MiB) copied, 1.99937 s, 201…

You should read all the topic, there are pros and cons.

[Ironclad14]

Creating symlinks doesn’t seem like a very good way in my opinion, it’s too workaround-y. Lets see what @codinghorror and @techAPJ will come up with.

[techAPJ (Arpit Jalan)]

There is no howto for this as of now, added on my list to look into it and write one.

[Ironclad14]

Thanks! The 20 gigs I have should be sufficient for now but it’ll be useful.

[sam (Sam Saffron)]

I did this for free codecamp:

volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log
  - volume:
      host: /var/postgres_data
      guest: /shared/postgres_data

Basically I keep /var/postgres_data in root and then mount /var/discourse/shared/ on the second volume, additionally I put docker containers in /var/discourse/shared/docker

There are many ways of skinning this, just keep postgres_data on the fastest drive and you should be good.

[thabbs (Thomas Abraham)]

Another solution on OS level can be to mount /var/lib/docker on a dedicated partition.

To do this, please, stop Docker, make sure that no process accesses /var/lib/docker and do the following:

Provide an additional volume. In this example it is /dev/sdx8.
Format it accordingly, i.e.

mkfs.xfs -n ftype=1 /dev/sdx8

or

mkfs.ext4 /dev/sdx8

etc.

If it fails in mkfs.xfs you might run a newer version of mkfs.xfs which has ftype=1 already as default. Just issue

mkfs.xfs /dev/sdx8

Stop Docker.

service docker stop

Make sure the directory is not used anymore:

lsof | grep /var/lib/docker

Prepare directories and mount the new volume:

mv /var/lib/docker /var/lib/__docker
mkdir /var/lib/docker
mount /dev/sdx8 /var/lib/docker

Move all contents there to:

rsync -aH /var/lib/__docker/ /var/lib/docker
# mind the trailing slash. It takes care to copy the whole tree to the right position.

Don’t forget to add /dev/sdx8 to /etc/fstab.

Remove the old directory if everything is fine.

rm -rf /var/lib/__docker

[dylanh724 (dh)]

If following the above guide, as a semi-newb, let me know if I got this right:

1. Assume mounted volume is at /mnt/someVolume/
2. Make an empty dir at mnt vol?

mkdir /mnt/someVolume/discourse/shared/

3. When you say ‘mount’, do you mean symlink?

ln -s /mnt/someVolume/discourse/shared /var/discourse/shared

4. Edit /var/discourse/containers/app.yaml …

volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log
  - volume: #KEEP THE SAME, right?
      host: /var/postgres_data
      guest: /shared/postgres_data

5. move docker containers?

mv /var/discourse/containers /mnt/someVolume/discourse/shared/containers

Then maybe stop/start again? Do I need to rebuild? Anything missing or need correcting?

[sam (Sam Saffron)]

This stuff is messy, I recommend you create a test droplet and try it out there, it’s the easiest way to answer all questions with zero risk

[dylanh724 (dh)]

Does this mean my summary interpretation is also on the right track?

I’m not sure if a test droplet is the best way to go, in my case. The IPs will be different, I’d have to setup a new host, and even if I did all that, nginx would be completely off… I’m pretty sure this would be even-more confusing. I’d probably be better off just making a backup before I change anything.

[Ironclad14]

About to try this on a fresh install. I’ll report back.

[Ironclad14]

Okay, this is my /etc/fstab

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/md3        /       ext4    errors=remount-ro,relatime      0       1
/dev/md2        /boot   ext4    errors=remount-ro,relatime      0       1
/dev/md4        /home   ext4    defaults,relatime       1       2
/dev/sda5       swap    swap    defaults        0       0
/dev/sdb5       swap    swap    defaults        0       0
proc            /proc   proc    defaults                0       0
sysfs           /sys    sysfs   defaults                0       0
tmpfs           /dev/shm        tmpfs   defaults        0       0
devpts          /dev/pts        devpts  defaults        0       0

When I do mkfs.ext4 /dev/sdx8 do I have to do it to /dev/md4? It’s my biggest partition.

Thanks.

[techAPJ (Arpit Jalan)]

Based on @sam’s suggestion I spun up a new droplet → installed Discourse → added a block storage volume → moved uploads and backups to that volume. Everything worked out fine. Here is the howto:

Move Uploads and Backups to DigitalOcean Block Storage

So you want to move all uploads and backups to DigitalOcean Block Storage? Before you do this, though, you might instead consider Configure an S3 compatible object storage provider for uploads. It is no more complicated to set up and provides some other advantages. Setup a Discourse instance on DigitalOcean Setup a Discourse instance on DigitalOcean using our official guide. Add a Volume to your droplet Add a block storage volume to your droplet. Note down the name of the volume. In my case t…