Can Discourse ship frequent Docker images that do not need to be bootstrapped?

Dev
1

There is this open issue (which I just closed) on Discourse Docker repo asking

Discourse should behave like a standard docker image
a docker user excepts to be able to run discourse without a special launcher.
to define multiple containers (redis, postgres, discourse,…), links and volumes, docker-compose should be used instead.
Docker Compose overview | Docker Docs

The question is somewhat misguided, but I would like to address a broader one of

Can Discourse ship frequent Docker images that do not need to be bootstrapped?

This question if far more complicated, so let me start here

Compose and launcher operate at different levels

Compose is a tool used for multi image orchestration. It allows you to “define” environments, bring them up and tear them down. It allows simple generation of base images from Dockerfiles (provided they are not local)

Launcher is a tool used for a wide variety of processes many of which have no parity with Docker Compose

  • We have a series of pre-req tests that ensure a system is up to the task of running Discourse (enough memory / disk space and so on)

  • We provide lightweight wrappers on top of docker exec , docker start , docker stop and docker rm, these wrappers ensure containers are launched with the correct configuration, automatically restart and so on

  • We allow rich semantics with generation of base images, this allows you to mixin support similar to the closed INCLUDE proposal for Docker.

  • Unlike Docker compose, Launcher only deal with configuration of a single container

  • We provide utilities like “cleanup” that allow you to clean up old orphan docker images

  • Our installation process was always meant to be “philosophy” agnostic. You can run Discourse as a single “fat” container including pg/redis and the web, you can also run Discourse in multiple containers using launcher keeping db and redis in other containers. Launcher reflects this.

At the end of the day … launcher and Docker Compose are compatible, you can generate the base images you want using launcher and bring up your environment using compose. The tools do very different things.

If you look at the images on your box you will notice an image called: local_discourse/app (derived from your yml file name), this image can be pushed to your own registry, composed using Docker compose and so on.

So let me try to address some of the bigger question here instead.

Why do we need to bootstrap?

Discourse ships 2 Docker images that you can use to generate your environment:

  1. SamSaffron/discourse_base , config: Our base image that include all our dependencies excluding Discourse code. This image includes the “right” version of Ruby, Node, runit, postgres, redis and the list goes on and on. It also includes the plumbing for running stuff that uses this image, taking care of providing infrastructure for reliable cron, log rotation, a reliable boot up and boot down process and so on. All our images are derived off this image

  2. SamSaffron/discourse , config : a simple image generated from our base image that creates the Discourse user and performs a “bundle install”.

The process of “bootstrapping” takes a base image (SamSaffron/discourse by default) and performs a series of transformations on it that prepare the image for execution. We did not use a Dockerfile, for this process cause the Dockerfile did not support the rich transformations we needed and generated lots and lots of layers.

In the context of a full container for Discourse the bootstrapping process will

  • Ensure postgres configuration is in place, configuration is dynamic and can be overridden by parent configuration. For example, this process will default [db_shared_buffers][7] in postgres config to “256MB”, however users may amend the app.yml file to increase this amount, which in turn will amend the physical configuration file in the bootstrapped image. The Postgres template also takes care of pesky things like postgres upgrade, which are 100% absent from the official postgres images.

Dealing with issues such as this is totally omitted from the default images:

docker run -it --rm -v /tmp/pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword postgres:9.3
docker run -it --rm -v /tmp/pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword postgres:9.4
LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.auto.conf"
FATAL:  database files are incompatible with server
DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.4.4.

  • Ensure redis is configured correctly, including ensuring that redis is always launched before the web comes up in “fat container” setup.

  • Ensure rake assets:precompile and bundle install is executed. Perform a series of transformations on our existing config (including setting up log rotation, setting up directories, permission, configuring NGINX and so on)

  • Apply specific user transformations. Want rate limiting? no problem. Want to use CloudFlare? Sure. Want to apply your own transformation? You have a rich set of hooks during the bootstrap process that allow you to apply your changes at the exact right spot.

In a nutshell there are 3 strong reasons why bootstrapping exists in the Discourse context

  1. Allow end user to override various settings that are written into files (such as transformations of NGINX config and PG config)

  2. rake assets:precompile needs access to a “specific” customer database and config

  3. To ensure plugins are able to amend application.js file and insert plugin code (which is minified)

  4. To ensure custom colors which live in the DB are taken into account when generating the application CSS file

  5. To run DB migrations

What if we shipped an updated bootstrapped image?

Internally when we deploy we use a “pre-bootstrapped” image as our base image and then simply run “assets:precompile” and “rake db:migrate” as our only bit of custom bootstrapping code. This allows us to mostly eliminate the need for an expensive bootstrap on every deploy. Images for our customers are shipped to a central private repo, we deploy the same image on multiple machines to load balance customers.

There is a great appeal to having a simple “dockery” distribution for Discourse eg:

docker run -d --link discourse_db --link discourse_redis --name discourse discourse:latest

This kind of distribution would allow us to easily apply updates without needing do a new bootstrap EG:

docker pull discourse:latest
docker rm -f discourse
docker run -d --link discourse_db --link discourse_redis --name discourse discourse:latest

Bang, you have an update with very minimal downtime and you did not need to bootstrap.

But …

  1. Who / what is going to migrate the database?

  2. Who / what is going to ensure assets are precompiled if you have custom plugins? How are you going to install plugins?

  3. How is this going to work in a multi machine setup? If I precompile assets on machine A so they include a plugins changes to application.js how can machine B get that exact change?

  4. How are we going to apply non ENV var settings

  5. How are you going to apply transformations like the cloudflare config or rate limiting?

Only way to get this going very reliably would be

  1. Introduce an intelligent boot process that through distributed locking can cleanly generate compiled assets and migrate. Convert all params we have now into ENV, on boot generate the conf files we need based on ENV.

  2. Trade around with your own custom discourse images just as we have today

  3. Heavily amend core to allow for a more intelligently composed application.js and a smarter way for doing application.css

This is not a zero sum game

We can slowly chip at some problems:

  • We can automatically build and publish Discourse base images for tests-passed, beta and stable, if done carefully this can help a lot, cutting down on precompilation and bundle install times drastically. We have to be careful about image count explosion though.

  • We can migrate some (or even all) settings from params into ENV and introduce a boot sequence that applies the settings. This will allow people to tweak a lot of stuff without needing a full bootstrap

  • We can investigate a system that allows us to install a plugin without a “full” precompile, and maybe even ship plugin docker images.

That said, this is a huge system and I would like to focus on our real and urgent problems first. Which in my mind are:

  1. Writing upgrade logic for pg from 9.3 to 9.5 (which is just around the corner) I want us to move to 9.5 this year

  2. Moving us to a multi image install by default, its super nice not to need to stop a site during bootstrap, only way of achieving this is with a multi image install. However, I do not want users having to muck around with multiple config files, so we need to think of a format that would enable this simply.

  3. Supporting another config format that is not yaml (probably toml). Our #1 support issue we see is “I mucked up my yaml file”, guess what … compose also has this problem.

41 Likes
2

Some of this is ‘configuration management’, and there are tools to help with that. For example, the image could use Chef Solo or Puppet to pull configuration recipes from some location on the host. This would allow config changes to be made ‘live’, without having to rebuild the container.

4 Likes
3

OK, but… when’s that going to get set up during the install process? What if there’s already a Puppet config on the host, how do you make sure you’re not stomping on it or sending it to machines that don’t want it?

1 Like
4

Use a Docker volume mount to put it into the filesystem space in the container, I suppose.

1 Like
5

Yes certainly, we use pups, an ansible inspired templating system for configuration deployment. The main reason I built it was for simplicity, Ansible/Chef/Puppet ship with 1000s of options, I wanted our system highly restrictive and also friendly to the bootstrapping process.

For example, stuff like this, would be hellish to do in many of the existing templating languages, cause they are not purpose built for this kind of stuff in mind.

They expect to execute in an already running environment.

I do worry though about taking a “Puppet” like attitude here and “automatically” applying config every time people edit yml, it feels to me like it would backfire on us big time.

For our own internal deployment we will eventually ship to a config is pulled from a central repo on boot, kind of approach, but that is not really applicable to most of the end users installing Discourse.

5 Likes
6

FYI: Some of the stuff you described above is done in phusion/baseimage (based on the official Ubuntu image), e.g. providing a proper init(-like) system incl. cron etc. You might want to check this out.

Without having analyzed the Docker file in detail, I suppose many of the “powerful” features you need can be done by copying a script into the image and executing it on every docker run, doing some checks if bootstrapping is needed and just going ahead and run the app if everything is set up. This will incure a cost on every launch of the container but that should not happen too often.

The bootstrap process would detect if the container has not been set up yet (i.e. default / missing config files) and immediately shutdown the container in that case. The “proper” config can then be loaded into the container via volumes or a custom, user-generated Dockerfile (based on the discourse image) so deployments are easier and do not require packaging and shipping a full image.

Some of the preliminary checks (like for system ressources) can still be performed within the launcher script (or moved to the bootstrap process within the container, I don’t see a problem with that) – disk size checks are IMHO unnecessary (at least before launching the container) since docker would complain if there is not enough disk space to download the image itself. But I would move most of the other checks into the container and have the launcher script be a simple entry point for beginners or lazies (like myself).

3 Likes
7

I am very aware of @honglilai work with base image and have been tracking it for about 2 years now :slight_smile:

I did talk a bit about issues with “bootstrap on boot” the biggest pain point is asset precompilation, which due to oj weirdness is non deterministic. Meaning we need to really, really, really generate a new base image. Considering that plugins may amend the precompiled assets we are forced to defer this base image generation to the customer.

I do want to move to more aggressive base image updates, I think it is good. Bootstrap on boot may be feasible for single machine setups. We may be able to have something that works in 2 modes.

Regarding resource checks, all of them exist based on real customer feedback and many open support requests.

3 Likes
8

I don’t see why this shouldn’t be done on docker run. No need to update the image for that, have it as a “boot” step (even if it takes time, but that’s manageable).

If you don’t want this on every run, maybe add an entry point (e.g. rebuild) that builds assets. The standard entry point can then perform a few checks and abort if it thinks a rebuild is required – the checks don’t need to be comprehensive, just to stay clear of the worst footguns.

1 Like
9

Because:

  1. Precompiling assets may take a minute and a minute on boot is not acceptable
  2. Precompiling assets is not deterministic, if you boot an image on 2 boxes and load balance you would be super hosed.
3 Likes
10

Ah, yes. But then the second entry point option would work, wouldn’t it? On a fresh install which still requires precompiling the assets, the regular entry point can detect that and abort if precompilation is needed. The launcher script then simply runs the rebuild entry point (with terminal output) before it launches the container.

1 Like
11

Sure, it is doable to have a pre-bootstrapped image, it’s just a rather big change I want to focus on my big #3 in my OP first

I get that people want to just kitematic add Discourse, but it’s a journey to get there.

1 Like
12

Just generally curious about this from anyone who knows the answer… Why is it not deterministic? The contents and timestamps of the assets must be the same (on the same image), so what differs? Is there a random number generator involved?

1 Like
13

oj gem is weird… but hey … its super fast so yay :slight_smile:

2 Likes
14

Oh wow, hit some limit and had to wait 20 hours before I could answer. Is that the Discourse default or specific to this installation?

Anyway, back to topic:

Just to be clear, I’m not talking about a pre-bootstrapped image. I forgot about the Kitematic use case for a moment, sorry about that. But yeah, for that use case a try-discourse image, which is the base image + precompile, is the way to go, I think. Just have the Dockerfile have an extra RUN that is the precompile, I suppose. With a git(hub) hook, the try-discourse image can be created automatically whenever a new Discourse version is released. But that’s not news to you, so I should stop :wink:. But having a more self-contained Discourse Docker image will simplify the try-discourse / Kitematic problem significantly.

1 Like
15

Working as designed: you’re heavily ratelimited in the first 24 hours after you create your account.

I don’t think that’s feasible to do on every commit (remember, you can follow the tests-passed branch) - the Docker images are pretty big. Probably better to keep the image updates to major releases + dependency changes.

(Also remember that the rebuild/bootstrap operation creates a new Docker image, just for you.)

3 Likes
16

We could potentially create a layer and autotag, its a bit tricky though to get this all going.

4 Likes
17

You can point the automatic image builder from hub.docker.com to either a git branch (e.g. tests-passed, stable) or a git tag (e.g. v1.4.0) and “save” it with a Docker tag (e.g. tests-passed, latest, 1.4.0, in the order defined above). Docker Hub will then rebuild the image when the branch or tag is updated. You could also write a small bot that auto-adds Docker tags in the Docker Hub when a new git tag (version) is pushed, so no need to create a new Docker tag on Docker Hub whenever a new version is released. Pretty neat.

So there is no need to create the image on every commit or push.

3 Likes
18

I figure I should try and explain my perspective, because this topic is basically at the root of me being very concerned about deploying Discourse in production and committing sysadmins to supporting it being highly available.

I think what worries me most is that there are assumptions, expectations, tooling, best practices and principles that generally apply to how people work with docker images, and you manage to break almost all of them. The system you have currently is surprising, which is rarely a good thing, and has added to the issue that I can’t assume anything you’re doing follows convention.

  • For a highly available system I need to set up Redis, Postgres and the web app in a way where they can be managed, monitored and preferably clustered.
  • For security, I really want to know how to make sure that the system is security patched regularly, locked down and only has the minimum required stuff installed. I need access to application logs, to know what’s in there and subscribe to relevant security advisories.
  • For quality, I want to know that dependencies like Redis which have slightly different behaviour when running multi-node are developed & tested in that configuration.

We have sysadmins, and to some degree the wrapping you have gets in the way of them being able to do their jobs effectively because they have to learn something new that’s not really got anything to do with those systems.

From my perspective, nobody should care if you have multiple docker images for each component of your service (this is not unusual or surprising), but we should care if you package everything into a single container, because it breaks things and behaves differently than the setup you recommend - the configuration of each service component has the ability to affect others. The fact that you say a multi-container setup is what people should be using points to this being how it should be developed too, and if you should be using an HA redis cluster in production, it’s great to be able to use the same setup in development.

Your “philosophy” agnostic design actually breaks the elements of the Docker philosophy that allow you to cluster applications in Docker Swarm, Tectonic, Kubernetes, Amazon ECS, Google’s Kubernetes, CoreOS… I just can’t buy the idea that you’re being philosophy agnostic when you’re breaking compatibility and standards.

I believe that your Ruby dependency leaks through to the host machine through “launch”. In the cases of the lightweight container OSes this means that you can’t use Discourse easily (since CoreOS doesn’t really support you installing things because it has no package manager).

When you say that launch performs tests, I would hope that you could do that by layering that on top of Compose (Compose the environment + a test container), or providing another image that exists only to run those tests. Launch can morph (in part) into a container that takes template files and produces config files (such as nginx config, shell script variables). You can make containers that exist only to run acceptance / integration tests, and a container can be as small as 10MB using golang.

Assuming that I understand what you pointed out about using a dockerfile, you noted that it creates a lot of layers. This can be true if you do a lot of lines, but I believe that if you simply bundle a shell script that wraps up a lot of smaller steps into a larger one and execute that, you only have a single layer. This method lacks the nice hashing on the line to determine when you’ve changed something, but I don’t think you’re really benefitting from that anyway. You could potentially version your source script by adding its content hash value to the filename, and then each change to it could force an automatic re-run of the transformation steps. See: http://blog.tutum.co/2014/10/22/how-to-optimize-your-dockerfile/

As I understand Docker Compose, service startup ordering is based on links and volumes, but can also be done by asking docker compose to start them up in the required order explicitly. I believe that what you need there comes down to a small batch script that orders the compose calls.

It’s not surprising (to me at least) that the upgrade process for Postgres does not run as part of the standard images. For one, I would be amazed if you couldn’t use the standard docker mechanisms to override the default CMD (Running containers | Docker Docs), and secondly the postgres images provide documentation on how to extend them:

How to extend this image
If you would like to do additional initialization in an image derived from this one, add one or more *.sql or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files and source any *.sh scripts found in that directory to do further initialization before starting the service.

In a clustered high availability setup I would expect a DBA to be managing the database, or in the case of RDS, this might be handled by AWS themselves. A lot of work is typically expended on 0-downtime database migration, and the ability to run multiple versions of applications side by side and rollback, particularly around schema changes and data migration. To me, baking this process in to the bootstrap feels like it makes a lot of assumptions about the environment.

By not having a single process entrypoint be the way that you run your container, you miss out on other things like containers stopping when the program crashes, and things like the docker logging driver. That makes your application difficult to integrate with centralized logging if we want to, and more complex to monitor.

While this is really quite personal as a view, I tend to think of any batch script that’s longer than 10 or so lines (and has logic in it) to be a sign of something bad - at that point you should use a scripting language. A 650 line bash script is practically a harbinger of the apocalypse.

For configuration management, it feels entirely wrong to be discussing Chef and Puppet (that’s just in this thread though), since these are generally speaking tools for reaching in and changing VMs. There is a reason that CoreOS and others are using etcd / consul, and it’s largely that for high availability clusters, services may have to subscribe to and change config on the fly. Both also support distributed locks.

A number of your templates are actually interactions with the nginx configuration. The normal way of running ruby in a container (iirc) is not to use nginx (which you would on a VM to get a decent level of parallelism) but to just use more containers. Pulling Nginx out seems like a good idea, especially since there’s a template for SSL configuration in there - if it’s your front facing layer it’s even more important for sysadmins to be able to configure it and ensure it’s patched separately and preferably network segmented away from direct DB access. We may be load balancing servers anyway using ELBs, HAProxy or something else, and while it’s nice to have a starter nginx config generated, I want my sysadmins to be able to manage it without learning your templating system or re-bootstrapping the application.

Decomposing the service components may cause a lot of the templating shrink, because you’re not needing to mess around with a significant amount of configuration on a single machine, and a lot of these things can be handled by much smaller modifications and extensions of standard images (thus removing a lot of the installation, and cleaning up what things are requirements for what). The overview of your application becomes much simpler, as the service architecture becomes clearer.

Note that none of this should preclude providing a simple “out of the box” configuration for developers, small installations and trials, but it becomes easier for sysadmins to manage as needed. (See: How to use Docker Compose to run complex multi container apps on your Raspberry Pi · Docker Pirates ARMed with explosive stuff). Fixing your bootstrap to just produce an image totally answers the “how do I make sure the same thing is running on multiple machines”, because that’s exactly what having a docker image is supposed to give you, by relying on version tags or a local private repository.

I hope that I’ve done a semi-reasonable job of explaining why I find some of the choices concerning, and why they potentially make it more difficult than it needs to be for our teams to take responsibility for and own a deployment of Discourse. I really want to like it, because it means we don’t have to build or fork a forum solution to make it fit, but I also have to consider the cost of operation and SLAs we can give.

39 Likes
19

Great feedback!

To be brutally honest, we don’t optimize for this in our open source facing installs because:

  1. The vast, overwhelming majority of open source Discourse installs are small or medium size and easily covered by the all-in-one container solution.

  2. Even a larger install is comfortably covered by a fairly simple isolated web / data container setup, so you can update / restart without interrupting service.

  • Our hosting business involves running Discourse at massive scale; while we are obviously in no way opposed to others running Discourse hosting businesses themselves, or otherwise hosting Discourse at massive scale, it does not exactly align with our business interests to make it one-button easy for someone to click a button and scale Discourse to a billion active users on a single site.

So there’s very little incentive for us to improve case #3, massive scale, because a) few need it and b) it’s a major source of income and survival for our business in the form of our enterprise hosting plan, where we are literally selling deep knowledge of Discourse as we wrote it in the first place.

That said, if you’d like to contribute effort towards making the open source story for case #3 better, we’re completely open to that.

15 Likes
20

An appropriate support contract priced competitively against running multiple enterprise level sites could be an option that creates a business incentive for you (above and beyond that I think it might make your own business operations more efficient). There are reasons like branding that companies might want to do that, but be put off by the potential cost of multiple hosted sites, and it’s a tried and true on-prem business model.

In terms of contributing, to be brutally honest, it would have to be a choice based on on business priority and the cost of developer time ;). Getting familiar enough with the codebase to be able to make good choices and know their consequences (thereby having a good chance of having the work pulled back to mainline) rather than give general advice would require an investment.

3 Likes