We are going to migrate our server and the discourse installation.
We are using a new server with btrfs file system.
I am making some tests in a test machine, I have copied all the files and installed all neccesary partes (nginx, docker, discourse itshelf).
I have tried with an ext4 filesystem and it worked OK.
But now, when I do the same with a btrfs formatted filesystem I get this erro when I try ‘launcher rebuild app’:
Your Docker installation is not using a supported storage driver. If we were to proceed you may have a broken install.
overlay2 is the recommended storage driver, although zfs and aufs may work as well.
Other storage drivers are known to be problematic.
You can tell what filesystem you are using by running "docker info" and looking at the 'Storage Driver' line.
If you wish to continue anyway using your existing unsupported storage driver,
read the source code of launcher and figure out how to bypass this check.
Obiously, docker info state it is using btrfs.
I have read in this forum that discourse has problems with some docker storage driver and that is why it refuses rebuilding.
Is there a way to change it to ‘overlay’ or other driver compatible with discourse and able to retrieve the files from the btrfs filesystem?
How should I configure docker?
It it posible to do that only for the discourse container and let the rest of docker configuration as default?
Both overlay and overlay2 are currently unsupported on btrfs or any Copy on Write filesystem and should only be used over ext4 partitions.
As Docker doesn’t support using the overlay2 driver on top of btrfs, it looks like the recommendation from the launcher tool is the extent of your options here. That is, continue running Discourse on a system where Docker is backed by ext4 or modify launcher to ignore the storage driver and hope for the best.
Commenting out (prepending # to) the exit 1 line in the following lines of the launcher script with your preferred text editor will achieve the latter if you want to take that route:
Considering “Other storage drivers are known to be problematic” though, I wouldn’t recommend doing that.
So if I understood it OK, docker is not able to use other alternative storage driver to access underlying system files, if you are using a COW filesystem like btrfs.
So there is no good solution, as discourse may have problems using btrfs storage driver from docker.
Reverting to ext4 is not an option, as all the migration was to be sure the data files were kept under the COW btrfs filesystem, and its ability to take snapshots, and keep data clean.
There is no point in using btrfs in other parts of the system, as its main objective is provide discourse forum access.
It is a pity because it would be great to have it running with the protection of a COW filesystem.
It is easier to use the --skip-prereqs flags. But using it in the productions system can be risky.
I have tried it in the testing machine and it works OK for now, but the problem seems to be in the long run.
Using --skip-prereqs skips many other tests which as you mention, introduces various risks when performing a rebuild. Commenting out that one line is no more or less risky than running on btrfs and it should auto-merge when launcher updates itself, meaning you can probably make the change once and mostly forget about it.
To be honest, that message was aimed at the old devicemapper that was a know issue for reliability.
Over the years we used aufs and most recently we use the overlay2 driver, without any problems. If you want to go with brtfs to give a try, please do a report here a few months in.
Thank you.
In the production server I am afraid of doing that.
If there are issues with the docker driver and it writes corrupted data, having snapshots, backups or whatever won’t help you if there is a crash.
If theres is some secure way of maintaining the data in a backup, I could try…
what kind of problems happened in the past?
But nowadays, this COW filesystems are very usefull. You can take snapshots, data is protected against corruptions during write or other fails, it is easy to add space or move devices…
It would be great if we can see it in discourse in the future.
May be I can help testing it. I have it running in a test machine.
The problem is that if I edit launcher file to add btrfs to the list of supported docker storage driver, when I run the script it complains that the script has been changed locally and will be overwritten by the remote version (from github).
How do I solve this?
What’s the exact message you’re seeing and at what point in the rebuild does it happen?
I just started up a VM I use for testing and modified the line, then rebuilt. At the point where launcher updates itself, it did the git pull and did a fast-forward merge as I expected, leaving the modification intact.
The version I was updating from didn’t include a change to launcher itself but I would expect that to work the same way as long as there isn’t a conflict, i.e. as long as that exit 1 line or very nearby lines aren’t changed in the repository.
Thank for your response and interest.
Let me tray to clarify.
I have changed the launcher script to include btrfs as one of the aceppted formats.
In the check_prereqs function I have changed:
if ! $docker_path info 2> /dev/null | egrep -q 'Storage Driver: (btrfs|aufs|zfs|overlay2)$'; then
When I tried first to make a launcher rebuild app, I received a message saying that launcher had been locally modified, and whether I wanted to continue downloading files from origin.
So I had to let it as it was, make the rebuild and change it afterwards to start the app.
But I have tried today to make a rebuild again, and it seems it detects the change but does not complain and the change is preserved.
I don’t know it something has been changed recently in launcher and I could have originally an old launcher that dit not make the reconstruct and now it does (after updating it yesterday).
I am testing it with btrfs and all seems to work OK, you can start and astop application, make the backups, everything works OK for now.
Docker seems to create btrfs subvolumes for the containers storage data when it detects it is running on a btrfs filesystem and uses btrfs storage driver.
So it seems that it makes some optimizations when it clones containers or takes snapshots via docker commands.
But I don’t know if the driver may be faulty in some way (it does not seem an experimental driver in docker, there are no advises about using it in btrfs or I could not find them) and whether it would be better (if possible) to change docker info in order to run it using overlay2 driver as if were running in a standard filesystem.
In theory it would be possible for docker to write its file and make operations on a btrfs filesystem without taking into account its capabilities (as btrfs is not different from other filesystem at the user level, for file i/o).
Any opinions or experiences using btrfs docker storage driver or configuring overlay2 driver to be used when using docker on a btrfs filesystem would be wellcome.
I don’t have any direct experience to offer but I would strongly advise against coercing docker into allowing the overlay2 driver to be used on top of btrfs. It’s explicitly unsupported and the overlay2 driver could be making assumptions about the filesystem that may or may not be true for btrfs, plausibly leading to various unexpected results. You really don’t want unexpected results at filesystem level.
Low level filesystem operations will be taken care of by docker and the btrfs storage driver. If that’s a mature and well maintained combination, which is not known-buggy as devicemapper was, there’s a good chance it will just work fine.
The reason btrfs isn’t supported in Discourse, as I understand it, isn’t because of some expectation that it will fail, it’s simply because they don’t have enough information to tell people that it will work.
They don’t have any (or enough) incentive to run their own servers on btrfs so the only way for them to get that information is from people in the community trying it out in production and reporting back after extended periods of time. That hasn’t happened yet, so it remains unsupported.
If you find yourself in this situation in future, you can always reset the change, update, then apply the change again before running launcher:
cd /var/discourse
git reset --hard
git pull
sed -i 's/Storage Driver: (/Storage Driver: (btrfs|/' launcher
./launcher rebuild app
So, I will not try to use docker’s overlay2 storage driver over btrfs filesyste, I will let docker use btrfs driver expecting it to work properly and not having any problem.
Here Docker Storage Drivers | Learn the Different Storage drivers of Docker they say it is the recommended aproach and officially supported for SLE linux, but recommended in Ubuntu.
Debian 10 and 11 support btrfs in the kernel with no modifications and support booting from a btrfs partition (only the UEFI partition should be of other type).
So I assume it is well tested.
Answer from Rafael seems to indicate that there is no special reason for not using it. Problems were with devicemapper and they put the request to use well known filesystems, probably in a time were there was no so much attention to btrfs or other COW systems.
I will give it a try.
I will report back my experience (good or bad) in its use.
For now it works smootly and let me change easily filesystem size, add devices, remove them, etc and gives me a confidence that the underlying data is correct and will keep with no errors.
The only precaution is with docker storage driver if is not tested enough, but it seems it has been widely use in SLE (which implements btrfs as defaults storage filesystem from long time ago).
I have to say that we have the production server running for 9 days over a BTRFS filesystem with no problem at all.
I had tested previously in test server.
And I have move the forum from place to a subvolume, added and removed disks and space from the filesystem, with no problem.
I will report after more time using it.
I am quite new to BTRFS and do not know it deeply, but it is not as difficult and it works as expected.
Well I have to say, that we have been running discourse in the btrfs filesystem for almost a month without any single problem.
It works smoothly.
The btrfs drivers of docker seem to work perfectly.
It would be great if some other people tests discourse running over btrfs and btrfs support could be integrated under discourse distribution.
We stop the forum for a moment, take the snapshot using btrfs (a couple of senconds) and run it again.
Then we make an external backup of the taken snapshot.
It seems to work perfectly.
I have restored the backup in another machine to test and it works.