Report on using discourse_docker on a CoreOS image

[julsevern]

I can happily report that discourse_docker works just fine on a CoreOS image. There was exactly one thing I had to modify:

localhost discourse # git --no-pager diff
diff --git a/launcher b/launcher
index bc7f204..07dca0d 100755
--- a/launcher
+++ b/launcher
@@ -16,7 +16,7 @@ local_discourse=local_discourse
 image=samsaffron/discourse:1.0.3
 docker_path=`which docker.io || which docker`

-docker_ip=`/sbin/ifconfig | \
+docker_ip=`ifconfig | \
                 grep -B1 "inet addr" | \
                 awk '{ if ( $1 == "inet" ) { print $2 } else if ( $2 == "Link" ) { printf "%s:" ,$1 } }' | \
                 grep docker0 | \

I had to use current CoreOS alpha release (431.0.0), because this is the first one with Docker 1.2.0. This release uses btrfs instead of aufs, but it doesn’t cause any kind of issues.

I’d like to follow CoreOS’ way of managing containers, maybe the correct way should be to maintain a private docker registry, and to pull it from there instead.

Thoughts?

[codinghorror]

Interesting

CoreOS

• claims 40% less memory use
• automatic full OS update scheme
• native / required Docker image support for all software installs
• optimized for large “cloud” provisioning of many servers
• read only OS images

I am curious if that 40% less memory figure is true! has anyone compared a 1GB DO Ubuntu image with Discourse, to a 1GB DO CoreOS image?

[sam]

This release uses btrfs instead of aufs

I liked this topic, but you just made me double like it.

aufs is a crutch, startup time of Ruby processes is heavily impacted. device mapper has always been flaky.

If you ever wonder why rails c takes so long, or rake assets:precompile, it is aufs

[sam]

Regarding memory use you should not expect huge across the board gains, the 40% figure is just relating to the “base OS” not the applications running.

Discourse consumes the vast majority of memory and mem usage is identical.

However, performance wise it is quite interesting.

btrfs means that we halve our boot time, so booting Discourse goes from 12 seconds down to 6.

Time to recompile assets (cached) rake assets:precompile

CoreOS (alpha 431.0.0)

real	0m38.563s
user	0m36.052s
sys	0m1.246s

Ubuntu (14.04.01 + Docker 1.2.0)

real	0m46.913s
user	0m40.888s
sys	0m5.576s

---

Time to start up rails time rake routes

CoreOS

real	0m5.790s
user	0m4.807s
sys	0m0.756s

Ubuntu

real	0m10.267s
user	0m5.945s
sys	0m4.179s

---

Time to recompile assets (uncached) redis-cli flushall && rake assets:precompile

###CoreOS

real	3m40.645s
user	3m26.428s
sys	0m2.381s

###Ubuntu

real	4m45.688s
user	4m38.128s
sys	0m8.702s

---

This doubling of boot time speed is most likely due to the usage of btrfs.

[julsevern]

Ha yes, it cannot reduce memory usage magically and the OS doesn’t have build and scripting tools whatsoever. The promised 40% is about the default not loaded OS footprint.

I had to increase the coreos KVM instance’s memory allocation from 1G, because of OOM errors in Discourse. However, it’s a good idea to run multiple dockers on the same image, for better memory utilization.

The other question is how we can leverage CoreOS’ cluster feature. The documentation usually refers to a kill - rm - pull - run sequence, while discourse_docker keeps the original container if available. Is there a technology which updates the docker image other than launcher rebuild?

[codinghorror]

we halve our boot time, so booting Discourse goes from 12 seconds down to 6.

I suspect this filesystem would also greatly increase upgrade times as well? Since that involves a lot of file magic.

[sam]

If I were running in a fleet I would run a central registry and just pull from it as needed.

Basically you bootstrap somewhere push to central registry. Then pull that image to the fleet and launch it using ./launcher start.

[maze]

I am curious if that 40% less memory figure is true! has anyone compared a 1GB DO Ubuntu image with Discourse, to a 1GB DO CoreOS image?

On a 1 GB CoreOS machine without swap, I had to cut down the number of unicorn instances to two. Also the update function (at /admin/upgrade) leads to out-of-memory situations. I have to stop the unicorns before trying to do that.

[codinghorror]

That’s the same as it is on Ubuntu. We recommend only 2 workers on 1GB per the official install docs, and we also recommend setting up swap to cover the upgrade case – upgrading chews through a ton of memory.

[nahtnam]

@sam Just saw the comparisons you made between ubuntu and coreos.

Do you think its worth switching from ubuntu to CoreOS?

[julsevern]

@nahtnam I’d wait until a coreos stable release contains a version of docker which is actually supported by the launcher script.

[sam]

I think we can get btrfs going in ubuntu which should be far simpler / safer.

[magnars]

@julsevern Five months later, were you able to run Discourse by following CoreOS’ way of managing containers?

[julsevern]

@magnars TBH I haven’t spent too much time on that It is still running well, but with only one CoreOS instance. I had to rebuild the docker image a couple of times already, which means 15-30 minutes of outage, but usually the update succeeds inside the container.

My problem with building and distributing an image inside CoreOS is you also have to at least separate the databases, and database upgrades can also cause you headaches.

[tsu]

Discourse installs without any modifications on CoreOS beginning with:

https://github.com/discourse/discourse_docker/pull/100/commits

Using CoreOS’ fleet is trivial but requires a small change which prevents that a different restart policy than below is set when boostrapping or rebuilding:

--- a/launcher
+++ b/launcher
@@ -319,5 +319,5 @@ if compare_version "1.2.0" "$docker_version"; then
   echo "We recommend you upgrade docker, the version you are running has no restart policies, on reboot your container may not start up"
   restart_policy=""
 else
-  restart_policy=${restart_policy:---restart=always}
+  : ${restart_policy:="--restart=no"}
 fi

… and here is an exemplary service file:

[Unit]
Description=Discourse for my EVE Online clan
ConditionPathIsDirectory=/var/discourse/shared/section8dot
After=docker.service
Requires=docker.service

[Service]
Environment="SUPERVISED=true"
WorkingDirectory=/var/discourse
ExecStart=/var/discourse/launcher start section8dot
ExecStop=/var/discourse/launcher stop section8dot
ExecStopPost=-/bin/bash -c "/usr/bin/systemctl stop docker-$(docker ps -a --no-trunc | grep -m 1 -F discourse/section8dot:latest | cut -d ' ' -f 1).scope"

[Install]
WantedBy=multi-user.target

[sam]

I don’t mind a PR to make restart policy optional

[ljpp]

How’s Discourse on CoreOS today? I am about to start yet another Discourse based project, to a somewhat high profile client (semi-celeb). The concept of automatic updates is very tempting.

Anyone using? Any lessons to learn?

[sam]

I recommend you stay on the beaten path, or in the case of high profile go for dedicated hosting.

CoreOS probably works, but what would you be gaining out of using it for a single site scenario.

The magic kernel upgrades still requires reboots afaik

[ljpp]

I don’t actually know why I mentioned the client profile - totally irrelevant.

This would of course be a learning experience for me, and the automatic update feature saves a bit of maintenance effort. Sure, the Ubuntu-way has been rock solid. Still it would be interesting to try CoreOS, at least for the development phase.

The CoreOS has several update strategies, regarding the reboot:

CoreOS