Is it safe to roll Docker from 18.01 back to 17.10?


(Rich) #1

Hey everyone,

Is it safe to roll Docker from 18.01 back to 17.10?

I have an almost identical issue to this one: Discourse crashed after backup - http 500

Our backups used to work fine, but now they fail every day and shortly after they fail, our entire site just hangs. Rebooting the server does cure it. Until the following backup the next day.

This problem only recently started, perhaps less than one week ago. Around the time I did a apt-get update (we are running Ubuntu 16.04.3 LTS ) and also on the same day I upgraded to Discourse v2.0.0 (currently on beta1 +65).

During the apt-get update the Docker was upgraded. I’m now running:

Docker version 18.01.0-ce, build 03596f5

And having read this post by @mpalmer ( Discourse web interface becomes unresponsive a few minutes after starting ) I think upgrading to Docker 18.01 may be the cause of all my issues. I can’t reply there as that topic is locked.

To give you an idea of the scale, our Discourse is relatively small, perhaps 5,500 posts or so with approx 120 users.

A db-only backup is approx 24MB and a full backup is 124MB - but the backups usually fail now, then the entire site stops working within a few minutes after that. A full reboot does cure it.

We run on a single DO droplet with 1GB/30GB:

Here are some details:

root@greyarro:~# sudo swapon -s
 Filename                                Type            Size    Used    Priority
 /swapfile                               file            2097148 649408  -1

 root@greyarro:~# free -h
               total        used        free      shared  buff/cache   available
 Mem:           992M        778M         72M         42M        141M         42M
 Swap:          2.0G        634M        1.4G

root@greyarro:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  4.5M   95M   5% /run
/dev/vda1        29G   13G   17G  45% /
tmpfs           497M  1.3M  495M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           497M     0  497M   0% /sys/fs/cgroup
/dev/vda15      105M  3.4M  102M   4% /boot/efi
none             29G   13G   17G  45% /var/lib/docker/aufs/mnt/840e6f4d9f2984c1d72ef416b7a64b19395914384de29ad89732b434e0599b5a
shm              64M  4.0K   64M   1% /var/lib/docker/containers/225ed791ef380a3dd272060a6c140d138df139bf696290af3624018633cccc61/shm
tmpfs           100M     0  100M   0% /run/user/0

A couple of days ago I recently uplifted all user-uploaded images to S3 so I’m not sure why that docker is using 13GB (perhaps that’s normal).

Anyway, my question is, while running Discourse 2.0.0 can I safely downgrade Docker from 18.01 back to 17.10 ?

Thanks for reading this far :blush:


Discourse crashed after backup - http 500
Discourse stops responding after the nightly backup has run
(Sam Saffron) #2

Should be safe to downgrade docker, start from a blank state to be extra safe


(Rich) #3

Ok, I’ll take a full backup at the weekend and try this on Sunday morning when it’s quiet.

Quick question about the downgrade process, Matt said:

  1. Stop and delete all running containers . In addition to this bug, Docker also changed some other things in the way that containers are run, and a downgrade while containers are still running will end in tears. If you’re only running Discourse on your server, you can just stop and delete the container with docker rm -f app (your data is safe, and won’t be deleted by this command). If you’re running other containers on the machine as well, you’ll have to figure out what to do.

I am only running Discourse on my DO droplet.

Could someone kindly help me out with the exact syntax I need to use, based on the details I posted above?

Would this be correct:

docker stop aufs
docker rm -f aufs

Or should I use:

./launcher stop app
docker rm -f aufs

Or should I be specifying the GUID?

Eg.

docker stop /var/lib/docker/aufs/mnt/840e6f4d9f2984c1d72ef416b7a64b19395914384de29ad89732b434e0599b5a

Or by ‘app’ does Matt mean the name of my app?

I also noticed @Tsirist didn’t mention removing anything in their post, so is this actually a requirement?

Sorry, in case you couldn’t tell, this is all a bit new to me and I really don’t want to break anything :blush:


(Bhanu Sharma) #4

You may not want to remove aufs

Instead use launcher destroy app

Make sure to have a recent known good backup. … Maybe even make a snapshot at digitalocean just to be extra sure.


(Rich) #5

Hi everyone, here’s what I did.

(Also related to: Discourse crashed after backup - http 500)

  1. Manuallly placed my site in read-only mode
  2. Performed apt-get update/upgrade (not really relevant buy hey, why not)
  3. Power off
  4. Take snapshot (digital ocean)
  5. Power on
  6. Ensure site is still in read only mode
  7. Remove my docker app: ( docker rm -f app )
  8. Stop the docker service ( service docker stop )
  9. Downgrade Docker to 17.10 ( apt-get install docker-ce=17.10.0~ce-0~ubuntu )
  10. Make sure Docker doesn’t get automatically upgraded again (see here)
  11. Start docker ( service docker start )
  12. Rebuild the app ( ./launcher rebuild app ) (not sure if this was required but thought I’d play it safe)
  13. Reboot the entire server
  14. :coffee:

Thanks for your help everyone, this worked a treat!! :+1: