Can't run ./launcher rebuild app - Docker storage driver is unsupported


(LT) #1

Hello,
I run a Discourse Docker image on a DigitalOcean’s Ubuntu 16.04.1 LTS droplet. Tried to update my SMTP settings in app.yml and an error occured while executing ./launcher rebuild app. The system returns:

Your Docker installation is not using a supported storage driver.  If we were to proceed you may have a broken install.
aufs is the recommended storage driver, although zfs/btrfs and overlay may work as well.
Other storage drivers are known to be problematic.
You can tell what filesystem you are using by running "docker info" and looking at the 'Storage Driver' line.

If you wish to continue anyway using your existing unsupported storage driver,
read the source code of launcher and figure out how to bypass this check.

Apparently, my Storage Driver is devicemapper , but I have no idea how to bypass the check or whether it won’t mess things up. Can anyone help me out? Thanks.


Launching ./launcher bootstrap app returns Your Docker installation is not using a supported storage driver on CentOS 7
Your Docker installation is not using a supported storage driver
Admin dashboard shows I am not updated to latest version, but I just updated?
Install failed on VM
(Matt Palmer) #2

That check recently got “fixed”, because previously it wasn’t actually checking the storage driver correctly. So, that’s why you’re suddenly hitting it now.

I’m a bit surprised that you’ve got an Ubuntu droplet running the devicemapper storage driver. Did you do anything special when you installed Docker on that droplet? I just did a test install of Docker on a fresh Ubuntu 16.04 droplet and I got Storage Driver: aufs out of the box.

If you’re only running Discourse on this machine, I’d strongly advise fixing up the Docker configuration to use another storage driver instead (aufs is our recommended option, but overlay seems relatively stable these days) by copying /lib/systemd/system/docker.service to /etc/systemd/system/docker.service and editing /etc/systemd/system/docker.service to add -s aufs to the ExecStart line in the [Service] section. Restart docker, make sure docker info | grep Storage shows aufs, and your next rebuild should work like a charm. Note that if you’re running other containers, they will be extremely confused about losing their backing storage, so don’t go doing this blindly in that case.


(LT) #3

Thanks a lot, Matt! I have no idea how I ended up with devicemapper after following DO’s step by step tutorials on Discourse and Docker installs, but somehow I did.

Thankfully, I was still testing Discourse and haven’t made it public for my website’s rather large community, so I’ve just deleted the droplet and started anew following Discourse’s own tutorial. Storage Driver is now aufs.

Thanks again!


(Perfectibilist) #4

Hey, I’m sorry but I was facing the same problem, did the changes in my

/etc/systemd/system/docker.service

restarted Docker, rebooted the system, nothing, can’t rebuild

Any help?


(Matt Palmer) #5

What exactly is wrong?


(Jay Pfaffman) #6

Fwiw, I did an install on a clean droplet and an update on an old one with no problems. (Except remembering to report it when they finished.)


(Perfectibilist) #7

I still get the same error, after I changed docker.service and rebooted.

Do I save a backup, and start a new VPS? :?


(Jay Pfaffman) #8

That’s probably the easiest. I do several installs each week with a stock 16.04 image and it just works.


(Matt Palmer) #9

What does docker info | grep Storage say? What about ps ax |grep dockerd?


(Allen - Watchman Monitoring) #10

I’m seeing this on some of my DO droplets. When i set it up a year ago, I didn’t knowingly diverge from the supported install.

docker info | grep Storage 
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: No swap limit support
Storage Driver: devicemapper

and

ps ax |grep dockerd
859 ?        Ssl    0:01 /usr/bin/dockerd --raw-logs
2876 pts/4    S+     0:00 grep --color=auto dockerd

Maybe related:

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

docker --version
Docker version 1.13.1, build 092cba3

(Allen - Watchman Monitoring) #11

following these steps, I end up with:

# vi /etc/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket firewalld.service
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// -s aufs
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

Neither restarting docker nor rebooting the box show the storage use being changed to aufs

Is there another step I’m missing to aufs?


(Allen - Watchman Monitoring) #12

I took a backup of the forum, and ran the upgrade to Ubuntu 16.

After that effort (which was small) I was rewarded with the following:

 ~# docker info | grep Storage 
Storage Driver: aufs
WARNING: No swap limit support

I don’t know if the warning is dire, but the discourse is back up and running, and on latest now.


(Andrew Waugh) #13

I believe that is an “informational” warning.
Have a look at:

As I understand it, docker allows you to limit the swap usage of a container, but since the discourse run command doesn’t do this (at least not for the standard install) the warning isn’t indicating a meaningful error.

See the bottom: the default (nothing specified) is: “There is no memory limit for the container. The container can use as much memory as needed.” This is the case for your discourse docker container.

This is similar to the THP warning you may have seen in your discourse logs (WARNING you have Transparent Huge Pages (THP) support enabled in your kernel")


(Tim) #14

FYI for anyone running on AWS EC2 instances, you have to use the overlay storage driver to get discourse up and running again. I accomplished this by editing /etc/sysconfig/docker-storage to add DOCKER_STORAGE_OPTIONS="--storage-driver=overlay" and restarted docker.


(Andrew Waugh) #15

Tim,

What AMI are you using, and when did you create it? I’m using the Ubuntu AMI and I followed the instructions in discourse/INSTALL-cloud.md at master · discourse/discourse · GitHub. I didn’t have to do anything not mentioned in the INSTALL-cloud instructions. This was the case in Dec - Jan.


(Matt Palmer) #16

I’m pretty sure @tim.smith is referring to the “Amazon Linux” distro, which is a badly mangled CentOS.


(Tim) #17

That’s correct. I’m using an AMI based on the Amazon Linux AMI.


(Alberto) #18

I have the same issue but i have 4 existing forum in devicemapper. When i change the storage driver i can’t see any containers. How i can migrate the forums from devicemapper to aufs??


(Jay Pfaffman) #19

That seems like a question for a docker forum.


(Matt Palmer) #20

The exact details would, indeed, be better suited for a Docker forum, but in short, you should be able to do a docker save of all the container disk images before you change storage driver, then do a docker load after the config change, and docker run them all again.