Bootstrap fails on CentOS 7

I’d like to add, this is being executed in the docker container… not on the actual host.

This is what leads me to think it’s something to do with the storage driver.

That sounds right. And broken. We gave up on another hosting service because of some other strange problem. (I think it had to do with networking.)

You might search for those terms with your hosting provider or docker and centos.

Not relevant in this case, self virtualized under ESXi. It’s a straight out of ISO install, with the exception of EPEL.

Trying with mainline kernel, will report back.

Same thing is occurring on mainline kernel, reverting to device mapper to see if outcome is successful… Definitely won’t be using that in production, however.

Update: Works with devicemapper, must be a problem between overlay and discourse.

I have tried with SELinux disabled, to no avail… as expected.

Devicemapper will corrupt your data over time in our experience, every time.

I have run overlay just fine in dev but not backed by xfs, only backed by ext4fs. It was certainly more quirky than running aufs, you got to be careful with inode counts.

I’ve heard that before (in regards to the inode counts). Device mapper is not something I will use in production.

I’m not sure exactly how to track this down any further. I will attempt to use Fedora 24 and Overlay tomorrow, although I highly doubt it will help, seeing as I was using a newer kernel on CentOS anyways.

Hi everyone,

I’m having the same problem here, trying to install discourse following the beginner install guide on an OpenStack CentOS 7 VM, getting exactly the same output & errors (http://pastebin.com/PJBcSJcH).

This is my docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.609 GiB
Name: discoursetest
ID: Y367:5ONL:JYSR:BACX:2ZD2:J3GF:VLS3:EFKA:YBYZ:NLVF:74AB:75SA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

I tried updating the kernel and still getting the same errors as @Luke. Docker info following:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.9.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.765 GiB
Name: discoursetest.novalocal
ID: Y367:5ONL:JYSR:BACX:2ZD2:J3GF:VLS3:EFKA:YBYZ:NLVF:74AB:75SA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Any new insights on this issue? The VM is sill deployed if extra info or testing is needed.

Any help or useful information about this problem would be greatly appreciated :slight_smile:

Docker file systems are, let’s say, complicated.

We choose AUFS and we guarantee you can host your data there and everything will work out.

We can’t afford to go after every distro+os fs+docker fs, so I can’t really help you on that.

1 Like

Useful information greatly appreciated!

Would be nice to mention that in the hardware requirements of the install guide to avoid time loss (and maybe maintain a compatibility table to track problematic & successful configurations tested by the community).

It is a limitation of your distro – you should be asking them why Docker is not stable on their platform, not us.

1 Like

Most probably it’s because the backing xfs filesystem was not created with ftype=1 option. This is very common if you let the OS installer format the / fs and you just have /var/lib/docker on /.

On newer docker verision (1.13?) with docker info you can see support d_type = false and that indicates your xfs was created with ftype=0 instead of the required ftype=1. This will cause weird issues if you use overlayfs on top of it.

See below issues for detail:
https://github.com/docker/docker/issues/27358
https://github.com/docker/docker/issues/9572

The solution:

  1. The easy way - just create another fs for /var/lib/docker with proper option if you have free space on disk. There are two options:
    a. Create a ext4 fs with default option and it will be fine.
    b. If you prefer xfs, create it like this: mkfs.xfs -n ftype=1 /path/to/your/device

  2. Or reinstall OS and during installation, make sure you manually create root fs with proper option. You cannot specify that option in the GUI/TUI installer so have to run the command by hand in the terminal provided by the installer. The command is mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE per RH document.

  3. Or change your current root fs option from the default ftype=0 to ftype=1. Unfortunately there is no online conversion way exists so you can only do a xfsdump/xfsrestore. Basically you need to prepare a new replacement root partition or disk, manually create xfs on it with proper option, then dump original root fs to the new one, and boot from the new root fs finally.

With any of above solutions applied, in docker info output you will see support d_type = true. Then you won’t run into the issue.

Edit in 2017/02/24: the symptom you may see with overlayfs on a d_type=false fs may be similar - strange errors during some very basic file operations, such as chown, chmod, rm etc… If you happens to run into such problems, and you find d_type=false with docker info, then above solution is for you.

9 Likes

Hi Krami i have a centos 7 and i change storage from devicemapper to overlay and when i launch the rebuild It send me error and is because I have ftype=0

[cevmf@server discourse]$ xfs_info /usr/lib/systemd/system/docker.service | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

I try to change like you said with:

sudo mkfs.xfs -n ftype=1 /usr/lib/systemd/system/docker.service

but without luck, can you please help me what i´m doing wrong.

That’s not how you use mkfs.xfs … Yo need to have some basic understanding of filesystem vs. files.

mkfs.xfs creates a new filesystem(means wipes existing one if there already is) on a partition, it is not meant to be used on a file.

Here is RHEL storage administration guide. You have to at least understand partition and xfs.

You may need to spend sometime on it before you start to play with your precious data.

4 Likes

Hi Kraml in the solution 1, suppousing that I have the partition ext4 I just have to do a new installation of docker there?, and uninstalling the other version of docker? without move nothing in /var/discourse correct?

You need to mount the new partition under /var/lib/docker. If your don’t have containers other than discourse, you can stop docker, rename /var/lib/docker directory(for backup purpose), create a new one, mount the new partition under it, and start docker again. Then you need to bootstrap discourse and get back the container. Your discourse data is not under /var/lib/docker so they won’t lose.

3 Likes

To use overlay the new partition must be EXT4 or XFS with the flag ftype enabled… EXT4 is the recomended option.

2 Likes

The issue described in this thread (permission denied) is due to SELinux, not ftype=0.

See Support official Docker packages on Red Hat/CentOS

2 Likes