Can't install Discourse with only 10 GB disk, run out of space

[hedgehog (Ali)]

You can select Ubuntu 14.04 with docker as your operating system.

[pl3bs]

Yea, I just did some checking and see that docker is now supported with the latest openvz kernels.

https://openvz.org/Docker_inside_CT
https://openvz.org/Download/kernel/rhel6-testing/042stab105.4

Not sure what your issue is. Sorry.

[hedgehog (Ali)]

@pl3bs: I tried your Dockerless installation script on the OVH-host but I got problems with it as well.
I filed this bug in your repo: https://github.com/pl3bs/dontdockmebro/issues/6

[pl3bs]

There’s nothing wrong with the script, it was tested by another user just two days prior.

94: read -p "Enter the name of your domain [ex: www.webeindustry.com] " domain;
95: sed -i "s/"www.example.com"/$domain/g" discourse.conf;

It seems you’ve skipped this part.

[hedgehog (Ali)]

I really had the wrong domain set up but I changed it the HOSTNAME in /var/www/discourse/config/discourse.conf rebooted but still get the same error.

[mshappe (Michael Scott Shappe)]

Crazy thought: can you please paste the output from

sudo dumpe2fs /dev/ploop10158p1

About the only idea I have left is that for some reason the block size of your filesystem is abnormally large. That would cause everything to appear to take up much more space…

[hedgehog (Ali)]

Can’t use dumpe2fs:

dumpe2fs 1.42.12 (29-Aug-2014)
dumpe2fs: Operation not permitted while trying to open /dev/ploop41512p1
Couldn't find valid filesystem superblock.

[mshappe (Michael Scott Shappe)]

OK, rather than trying to guess what the actual filesystem type is, can you please paste the contents of your /etc/fstab?

[hedgehog (Ali)]

/etc/fstab

proc /proc proc defaults 0 0 none /dev/pts devpts rw 0 0 none /dev/shm tmpfs rw 0 0 cgroup /cgroups/blkio cgroup blkio 0 0 cgroup /cgroups/memory cgroup memory 0 0 cgroup /cgroups/devices cgroup freezer,devices 0 0 cgroup /cgroups/cpuset cgroup cpuacct,cpu,cpuset 0 0

[mshappe (Michael Scott Shappe)]

Hm. OK. Apparently OpenVZ makes this sort of thing more opaque than we might like, since that tells me nothing about /dev/ploop* How about cat /proc/mounts?

[dandv (Dan Dascalescu)]

My Discourse instance has run out of 25GB of space (Linode 1024 VPS). The only files over 100MB were 7 backups, 225MB each, and one 700MB nginx log file. Hardly adding up to 2.5GB.

When I ran ./launcher cleanup, more than 10GB were freed up.

Clearly there’s a leak somewhere. I’d be curious how much disk space Docker takes up on other folks’ similar configurations. Right now, I have

root@localhost:~# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/root                                   24G  7.4G   16G  33% /
devtmpfs                                   494M  4.0K  494M   1% /dev
none                                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                                       100M  340K   99M   1% /run
none                                       5.0M     0  5.0M   0% /run/lock
none                                       496M     0  496M   0% /run/shm
none                                       100M     0  100M   0% /run/user
/dev/mapper/docker-8:0-41527-352bef0[...]  9.8G  1.5G  7.8G  17% /var/lib/docker/devicemapper/mnt/352bef0[...]

[gerhard (Gerhard Schlager)]

Usage of devicemapper is not recommended. I don’t think that has changed in the last few months.

https://meta.discourse.org/t/why-is-docker-devicemapper-taking-so-much-hard-drive-space/34378/11?u=gerhard

[codinghorror (Jeff Atwood)]

If you are using device mapper, you are in a world of pain and suffering. Strongly not recommended. We block install on device mapper for a reason.

[dandv (Dan Dascalescu)]

Well, I never did anything intentional or specific to end up with devicemapper. Never saw a warning, let alone been blocked from installing.

I installed Discourse around June 2015 (I think - is there a way to tell specifically?), so maybe devicemapper was the default back then for Docker on Linode kernels, after Docker dropped the aufs requirement? I’m running Ubuntu 14.04.3 LTS (GNU/Linux 4.4.0-x86_64-linode63 x86_64) and have upgraded the Linode to KVM from Xen.

1. What exactly does this “world of pain” entail? Is it worth restoring a Discourse backup to an instance running on aufs?
2. Should there be some guidance in the 30-minute installation guide about aufs? Right now it assumes DigitalOcean, but folks may want to install on other hosts, for whatever reason.

[fefrei (Felix Freiberger)]

What exactly does this “world of pain” entail? Is it worth restoring a Discourse backup to an instance running on aufs?

The short version: It will work fine until it suddenly corrupts the file system; and then demons fly out of your nose, circle back behind you and strangulate you. Only one of these is an overstatement

Here are a few memorable quotes from the world of pain:

We had some very nasty support issues that were incredibly hard to troubleshoot.

What is the best docker storage backend to use on a VPS?

Never, ever use device mapper. That is a one way ticket to corruption town, sooner or later. It is buggy as hell and has been for a year despite Red Hat saying “it works now” over and over.

What is the best docker storage backend to use on a VPS?

I don’t know but you can ask the support customer whose corrupted install on Centos (and thus devicemapper) we just had to recover via a backup…

[sam (Sam Saffron)]

@dandv

Should there be some guidance in the 30-minute installation guide about aufs? Right now it assumes DigitalOcean, but folks may want to install on other hosts, for whatever reason.

I am pretty sure launcher warns about non aufs these days.

https://github.com/discourse/discourse_docker/blob/master/launcher#L106-L115

[dandv (Dan Dascalescu)]

I am pretty sure launcher warns about non aufs these days.

Not a peep actually. I’ve just installed on a Ubuntu 14.04 LTS Linode. Here’s the full log.

http://pastebin.com/68wRhftD

[sam (Sam Saffron)]

What is docker info returning?

[dandv (Dan Dascalescu)]

root@localhost:~# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.10.1
Storage Driver: devicemapper
 Pool Name: docker-8:0-98371-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.075 GB
 Data Space Total: 107.4 GB
 Data Space Available: 19.82 GB
 Metadata Space Used: 1.974 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-x86_64-linode63
Operating System: Ubuntu 14.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 991.7 MiB
Name: localhost
ID: WYHE:UE4G:DVLD:BQNI:2QRX:WTK6:ZDRP:MQOM:QWAP:F2C5:7N7D:CEXH
WARNING: No swap limit support

[sam (Sam Saffron)]

So odd … any idea why that test Can't install Discourse with only 10 GB disk, run out of space is not running, try putting some echo commands in launcher to see if its even hitting it?