Docker Upgrade Removed AUFS Storage Driver, Killed Discourse

Product Support Self-hosting

AI-generated summary

A user running Discourse on Ubuntu 20.04.6 LTS encountered an issue after a system upgrade: Docker removed the AUFS storage driver, preventing the forum from starting. The user shared debug output confirming AUFS was deprecated and no longer supported. pfaffman reassured them that data is safe in /var/discourse/shared and suggested rebuilding the app if Docker works. The user confirmed Docker wouldn’t start with AUFS but could start after switching /etc/docker/daemon.json to overlay2. They asked whether rebuilding via ./launcher rebuild app would preserve all data and migrate it to Overlay2. pfaffman affirmed it should work without data loss. The user confirmed success — the forum was restored. They then considered upgrading Ubuntu to 22.04.3 LTS, to which pfaffman advised caution, recommending instead to migrate to a fresh 22.04 (or even 24.04) image using rsync to avoid upgrade pitfalls and downtime, linking to a guide on migrating Discourse via rsync.

1

I run a Discourse forum installation on Ubuntu 20.04.6 LTS. The Docker image uses AUFS for storage.

Today, I installed upgrades via sudo apt upgrade. Now the Discourse forum is no longer accessible, and it appears to be because a Docker upgrade removed the AUFS storage driver.

I see plenty of guides to configuring Docker to use Overlay2 instead, but I don’t see any guidance on how to migrate my existing Discourse installation to use Overlay2, especially now that I can’t start Docker to mount my AUFS volumes.

When I execute

sudo dockerd --debug

The output is

INFO[2024-05-15T19:49:04.327570182-04:00] Starting up
DEBU[2024-05-15T19:49:04.332295421-04:00] Listener created for HTTP on unix (/var/run/docker.sock)
WARN[2024-05-15T19:49:04.340758649-04:00] failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously
DEBU[2024-05-15T19:49:04.427688615-04:00] Golang's threads limit set to 13680
DEBU[2024-05-15T19:49:04.430903802-04:00] metrics API listening on /var/run/docker/metrics.sock
DEBU[2024-05-15T19:49:04.554881397-04:00] Using default logging driver json-file
DEBU[2024-05-15T19:49:04.555543790-04:00] No quota support for local volumes in /var/lib/docker/volumes: Filesystem does not support, or has not enabled quotas
DEBU[2024-05-15T19:49:04.563660483-04:00] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2024-05-15T19:49:04.567109514-04:00] Error while looking up volume                 driver=local error="volume not found" volume=
DEBU[2024-05-15T19:49:04.567813165-04:00] Error while looking up volume                 driver=local error="volume not found" volume=
INFO[2024-05-15T19:49:04.574217050-04:00] [graphdriver] trying configured driver: aufs
DEBU[2024-05-15T19:49:04.574500003-04:00] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2024-05-15T19:49:04.574603992-04:00] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2024-05-15T19:49:04.578694693-04:00] Cleaning up old mountid : start.
DEBU[2024-05-15T19:49:04.579329989-04:00] Cleaning up old mountid : done.
failed to start daemon: error initializing graphdriver: [graphdriver] ERROR: the aufs storage-driver has been deprecated and removed; visit https://docs.docker.com/go/storage-driver/ for more information: aufs

How can I restore my Discourse forum without losing any data? Can I somehow restore the AUFS storage driver and then migrate to Overlay2 storage?

2

You won’t lose data, it’s in /var/discourse/shared.

If docker works, just rebuild and you’ll be fine.

3

Unfortunately, Docker stopped working after the upgrade. You can see the error in the final line of the output I provided.

I can get Docker to start if I change the contents of /etc/docker/daemon.json from

 {
     "storage-driver": "aufs"
 }

to

 {
     "storage-driver": "overlay2"
 }

However, my Discourse forum still doesn’t work after making that change.

Are you saying that it will work, and maintain the users, posts, and rest of the configuration and historical data if I execute the following and restart Docker?

cd /var/discourse
git pull
./launcher rebuild app

Will rebuilding Discourse in that manner migrate all the forum data to Overlay2 storage?

4

I expect that will work. I can’t imagine a way that you’ll lose data.

5

You were right, apparently. After changing the storage driver in /etc/docker/daemon.json to overlay2 and executing ./launcher rebuild app, the forum seems to be back up and running.

Now I wonder if I should tempt fate and do an Ubuntu release upgrade to 22.04.3 LTS. :slight_smile:

6

I would wait another bit and create a new 24.04 image and migrate to it rather than do an upgrade, or just move to 22.04 now. I don’t trust how much stuff doesn’t get upgraded on an upgrade. Plus, you avoid the downtime and if anything goes wrong your old server is still working.

Move a Discourse site to another VPS with rsync