It’s absolutely essential for our many self-supported Discourse installs out there that they have an easy one-click updater.
I can’t recall the last time (or even any time) a self-supported install has complained that docker-compose wasn’t used.
That’s the highest priority here, it’s critical to our project and business – “purity” of whether docker-compose was used does not even register in the scheme of things. Not even a little.
Sorry.
I just don’t see the business driver to go and throw away our old tooling and move to new tooling.
If we did this
We have to worry about a giant migration to the new system, if we are to somehow automate it probably in the order of a month of engineering.
We have to rebuild all of our tooling which is probably months of engineering
We need to deal with the “log rotation” problem which is unsolved in Docker compose, who knows maybe compose another container to rotate logs.
We need to accept an uncompromising design that requires at least 3 containers to run every Discourse instance (probably 4 or maybe 5) with all of the caveats that come with that.
We have to solve the plugin problem which is a nightmare with no bootstrap process
And what problem are we solving? Make it easier for people to debug and contribute to “discourse docker”? We are not seeing a particularly big pain point there and getting plenty of contributions.
I just am not seeing what problem this is solving. It is not solving. “Make it easier for self installers”, arguably it is solving “Make it easier for self installers that are seasoned docker compose users.”
However, the vast majority of our self-installers don’t even know what docker is.
+1 to this post.
I know it is hard to change the existing architectures, so I understand @sam’s headaches very well. I love the Discourse in the way that how the modernized web forum software should be looked like. But in addition to the front-end innovation, it would be much helpful for the market and the developer ecosystem if it be much more aligned with the modern infrastructure practices, including Docker compose, Kubernetes, AWS ECS, mesos, etc.
Docker is about making a standard container for software build, shipment, and deployment. The Docker ecosystem, including commercial cloud vendors such as AWS, make the toolkit with that fundamental assumptions.
Another perspective is to decouple build/packaging job from deployment. Blue/Green deployment strategy is a good example. It is not recommended to run software upgrading/packaging on the production server directly. Instead, people usually launch a new version of virtual instance with the latest version software, and runs tests agains it. When the new version failed to test, then deprecate the instance and no production service would be impacted. If the new version passes the validation, then roll out the new version into new virtual instances, and shutdown old-versioned virtual instances. This is what Docker community say about immutable images.
As an example, I had a series of troubles after upgrade Discourse versions. For somehow reason, the new build was failed and I wanted to go back to the previous deployment state but couldn’t. It is because the Discourse builder tool already changed my production images. (thankfully, I did AWS AMI snapshot and could launch new instance with the previous AMI)
So again, thank you for making this incredible forum software; it showed how the modern front-end software should be looked like. But also I want to say there is big opportunity to become an excellent role model of modern infrastructure and software management using Docker industry standard.
The thing I find frustrating is that we are already aligned with compose/kubernates etc.
Once you bootstrap an image you can push it to your local registry. Then you can use standard tooling to launch it. In fact, that is exactly what we do internally.
Thing is… this is completely under your control
./launcher bootstrap web
docker tag local_discourse/web my/discourse:1.1.0
Does local registry mean a private registry or a local registry hosted on one of the servers?
My Discourse app is not tied to a fixed number of instances. In more details, as an example, I use AWS for hosting, and use AutoScaling groups to automatically scaling out and in according to the traffic. So my thought is to push “ready-to-run” container image to a central repository, and let my new EC2 instances fetch the latest Discourse Container image from the repo and run using standard tools such as docker compose or AWS EC2. Is it possible as of today?
Once you tag an image, if you tag it correctly you can then push it to your docker hub account (which is remote) or to a local registry
So it really depends on how you have stuff set up, but simplest is to register an account on docker hub and push the image there, then you can autoscale as you wish.
100% possible today
That makes sense. But again, I need to install launcher app into the new virtual instances and run that command; even if the image is pulled from a central repo (which is good), I still need to rely on the launcher utility to make it up and run.
What I see here is that there are many resources involved into managing launcher utility, which basically pulls latest git, build, do package validation, raking, and deploy/run/stop. The idea of Docker ecosystem is you don’t need to invest your resource on reinvnting wheels, especially deployment jobs. Why don’t you let launcher focus on package building and validation tasks and utilize Docker ecosystem tools for deployments? I personally believe your hosting business can seek more optimization opportunity from the operational cost and efficiency in the long run.
PS1. Oh btw, regarding the Docker compose, it is not necessary to break an image into multiple ones. I use Docker compose even when I manage only a single image because it is way more easy for me to manage. In fact I prefer Discourse’s single container architecture. The key idea of the compose is that I can rely on industry standard tools for the Docker shipment/deployment which is huge benefit. The docker compose tools are publicly accessible from Linux yum/apt repo, whilst the launcher is not.
PS2. Regarding the logs, although I personally don’t care, there are Docker container log management solutions. Splunk supports Docker logging, and Datatog is another good but affordable one.
No, what @sam is saying is that you only need the launcher utility on your build server; it runs once per deploy.
Thanks @riking. Can you elaborate a little more? Let us say I have a brand new virtual instance, which does not have any Discourse packages. And say I downloaded my latest version of my Discourse app image from my Docker hub which is tuned for personal use. Now I have latest Discourse Docker image on the new virtual instance. Can you describe how I should launch that image and start the service? Can I simply run docker run <image id>; or do I need to download launcher utility and execute launcher start app? My point is that I prefer docker run approach over relying on launcher for deployment.
All launcher start app does is turns conf file to a docker run command. In fact it outputs the docker run command it runs, you can cut-and-paste that 
No need to have launcher or anything installed on the box to run a Discourse instance. Launcher is simply a convenience.
Oh… I thought there are some magic happens inside the launcher but was not… Thanks for the clarifications! Okay, then I get more clear picture! Probably I can make a script to parse the docker command and generate Docker Compose yaml, so that I can use it for easy deployment.
Then… hmm… Discourse already has the good support… wonder why this lengthy thread began at the beginning… 
Just because it does not provide the Docker Compose yaml by default…? Anyway thanks for helping to understand.
Not that I have any serious complaints about the current monolithic image approach. In fact I am fairly convinced with the rationale given in this thread. I know that not taking the isolated container orchestration route is a compromise, but the consequences are well thought.
However, based on @jongnam’s last comment, I think it would be really cool to add a couple features in the launcher start app command:
--dryrun flag to only output the generated docker run command and not executing it automatically--compose flag to output a minimal docker-compose file on STDOUT which is nothing but a reformatting of the generated run commandThe latter would allow people to manually add more services (such as a local SMTP server), defining private network, or adjust certain parameters.
absolutely great suggestions, I would add that you also want a --stack so you can output the docker stack format as well which is around the corner.
PR totally welcome.
In case you missed this:
Thanks for the pointer. That was a fantastic read as I am still exploring stuff around Discourse. I wish, there was a built-in SMTP server solution as well. Currently, one has to use external providers or set it up manually.
I can give you an outgoing SMTP server image in no time at all, but it won’t help you much, because 99.9% of the work in getting outgoing e-mail setup correctly is in DNS (rDNS, SPF/DKIM), and in managing all the other crap surrounding mail delivery (like periodically checking blacklist aggregators and requesting delisting). That’s why we recommend an external mail relay for outgoing e-mail, because that is a lot of fiddly, specialist work that very few people want to deal with, and it isn’t something we can automate away. Contrariwise, if you’re dead set on running your own outgoing mail, setting up the MTA is the easy bit, so there’s not a lot of motivation to make the easy bit easier when the many hard bits are going to stay hard.
When I started running email servers under Linux, I had to port Sendmail myself, as no one else (who made the code available) had done so. I’m pretty good at it. When I finish moving off the server that I set up five years ago, I won’t be managing any mail servers that send mail. It’s just too freaking hard for all the reasons that @mpalmer outlined. The likelihood that whatever IP you get assigned is on some blacklist is insanely high. Even the email services like Mailgun and SparkPost have difficulty staying off the blacklists.
I’m not entirely convinced just how easy receiving mail is anymore, but I haven’t yet given up on that, and having Mailgun receive mail and then forward it to Gmail is annoying. I think I will soon add a $200 Discourse + Let’s Encrypt + direct incoming email install offering along side of my standard $99 Discourse Install.
Thanks @mpalmer and @pfaffman for insights. I think I am aware of these issues as I am also managing a hosted mail server (Exim4) for our current sites that are not on Discourse yet. I am certainly not denying the issues you people have described and actually I read such things here before as well, because built-in SMTP server seems like a well-requested feature (for me it was not a request, but a wish instead).
However, as far as my understanding goes, getting the IP address delisted from various blacklists is the real pain. Otherwise DNS records are not an issue and we end up doing that anyway even if we use third-party SMTP services.
Third party e-mail services will usually help you get your DNS settings correct, with checking tools and feedback. They also (especially for paid services) provide tech support. All of that would come back to here if we provided some sort of outgoing e-mail container.
At any rate, if you’ve got a mail server you’re happy with already, you don’t need to use a third-party mail service, you can just route the e-mail through your existing MTA.