Installing Discourse for macOS Development Using asdf and docker-compose

I don’t think that it’s the image that’s the problem. The guide warns that it is “much slower than a native install on macOS” and I found that to be true. My guide is really designed more as an alternative to the development on macOS guide but that tries to avoid global installs whenever possible.

Can you elaborate on this comment? Bitnami already has containers available for postgresql 12 and their container repositories are actively updated. There are alternatives to Bitnami, but I’m just not sure where your FUD is coming from…?

Any use of Bitnami is completely unsupported here. This isn’t Fear, Uncertainty and Doubt, it comes from several years of users coming here asking for help when their Bitnami installations of Discourse run into problems. Any Bitnami support topics are #unsupported-install.

As we routinely see issues with those packages, and as they are created and maintained by a third party, we can’t offer any assistance when said problems arise. No third party packages recieve support

You’re going to need a representative dev environment to ensure that things will work in a supported live install, Bitnami isn’t the way.

3 Likes

But this isn’t a Bitnami installation of Discourse—it’s a native installation of Discourse that uses Bitnami containers to provide postgres and redis services.

Can you point out where some of the differences might arise? For example, is Discourse using a custom postgres or redis configuration that could cause different behavior? If so I didn’t see any indication of that in the previous macOS setup guide, although perhaps I missed it. Nor are any such differences reflected by the test suites, which all run fine in this environment.

There’s just no reason not to use the postgres and redis installation provided by Discourse.

Can you explain what you mean by this?

In a development setup for macOS brew is used to install postgres and redis for Discourse to use. So what is the postgres provided by Discourse?

I’m struggling to see a difference between doing that and using a container that has the exactly same version running inside. Except the fact that I don’t have to perform a global install of those services and can easily stop and start that development environment as needed.

Right, but for reasons, you want to use a docker-based postgres and redis instead

Correct, yes.

I’m a bit confused about why I’m getting so much pushback here. I explained my reasoning above. I’m sure that the other approaches to setting up development environments have worked for many and will continue to. But using Docker compose to install the services needed for a specific development environment isn’t, like, a super weird thing to do—it’s actually exactly what docker-compose was designed to do.

By using a docker-compose.yml file you can pin the postgres and redis versions so that all developers are using the exactly same version, rather than just whatever brew happens to install on that particular day. Plus, as I’ve already pointed out, other people may need other versions of postgres or redis for other projects. Or just want an easy way to start and stop their development environment when needed, so that you don’t have a copy of postgres and redis running all the time in the background.

Again, this just isn’t a weird thing to do. My goal was to just share some helpful tips in case others want to set up a development environment in a similar way. YMMV, but I don’t think that I’m telling people to do anything dangerous here. I figured at least a few other people might prefer an approach that doesn’t require a completely new package manager or modify global settings (like changing the default ruby version).

Thanks! Is this configuration system used during development? I was under the impression that it was not.

It’s because you’re introducing yet another way to “solve” the “how to set up a decent development environment” problem. Who’s going to support it? Are you going to answer all questions that anyone on the planet asks about it? Are you going to continually upgrade your stuff so that it continually matches all of the versions for everything?

But using PG and Redis in a container seems like a pretty good idea to me, but if you’re going to do that, why not do it the way that is already supported?

By using the template that Discourse uses, you can pin Postgres and Redis versions so that everyone using Discourse is using the same version.

And, you can’t use docker-compose to do much else with Discourse, so it’s a tool that lots of people who come to Docker because of Discourse are likely not to be familiar with.

Correct. After pointing out quite reasonable reservations about the other two approaches. One being far too slow and the second not exactly following best practices for how to isolate different versions of tools and services from each other.

I’d be happy to try :slight_smile:! Bring 'em on…

I can prepare a PR to add the .tool-versions and docker-compose.yml file to the repository. There are really only four versions to maintain: ruby, yarn, postgres, and redis. I’ll assume that you have the supported versions or version ranges of those tools noted somewhere in the repository, and so it will just be a matter of keeping .tool-versions and docker-compose.yml synchronized across major updates.

The biggest thing that would need support from your end would be the changes to config/database.yml, although there is already precedence in those files for using environment variables to override certain database connection options.

Again—running everything in a container proved to be very, very slow on my machine. This way the core Discourse rails process runs natively. I’ve been using this for plugin development for the past 24 hours and it is much more responsive. Meaning that I actually got some work done :slight_smile:.

IIRC Docker compose comes bundled with Docker. It’s not a hard tool to use: docker-compose up when you start and and Control-C when you are done. You could easily add this to a script similar to d/unicorn if you wanted.

Keep in mind that the alternative native install on macOS has users run a shell script that uses brew to install whatever current versions of postgres and redis happen to be available at the moment, and then also starts them for good measure where they’ll run in the background forever after I (immediately) forget about them or don’t even notice what the setup script did. So the alternative to Docker compose is remembering to both brew services (start|stop) postgres and brew services (start|stop) redis. To me that seems equivalent / worse than using docker-compose.

Right, so it does make total sense to me to use the template I linked to run PG and Redis in the very same container that thousands of production Discourse sites are using. My recommendation is that if you’re going to run PG and Redis in a container, to use the one that Discourse provides. The further you step away from the Official way of doing things the more chances there are for hard-to-guess edge cases.

Fair point!

FWIW, I don’t use a Mac so my comments are fairly high-level.

I get that.

However, I would like to point out (again) that your supported macOS development install does not run postgres or redis in a container. It doesn’t even pin the versions of these services installed by brew, which I suspect is possible. What, if anything, is the difference between that setup and mine? I feel like here you’re holding my approach to a higher standard than your official macOS development instructions, which again can end up installing any old version of postgres and redis and AFAIK also doesn’t use the configuration fragment you linked to.

I guess I’m not sure what to do with this template. AFAIK that snippet is merged with a bunch of others and used as part of a fairly involved process that builds a container that includes everything Discourse needs to run: including the parts I want to run natively.

Is there a way to build an official Discourse container that only contains postgres and redis? If so that would be great!

At some point I’d be happy to experiment with a setup using the official Discourse container. I bet that it would be possible to get that to work… although some tweaking might be required. (For example: Discourse seems to want to connect to postgres by default using a named socket, rather than over a port. So is the database in the container even listening on the default port? I’m not sure.)

My goal here was to be helpful and try to provide an alternative development path that might appeal to some, but not all. I’ve made it very clear how my instructions differ from the standard install process and why. All of the test cases pass in my environment. My goal with the original post was to offer something helpful to the community, but after the amount of sniping its received I doubt that it’s accomplished that.

Yes. Edit the file that I linked and run

./launcher rebuild data
1 Like

This is my biggest issue here.

Our docker dev setup is slow because Docker volume performance is bad on Mac

So if somehow your docker setup is fast I want to know exactly why, cause the exact same fix can be applied to the official setup

8 Likes

Not to add more confusion but I’ve been using a non-docker setup by installing all the deps manually with my macos system and didn’t have to look back. I have been doing discourse development almost everyday for the past 9 months or so.

4 Likes

Your rails application does a lot of file IO, particularly on startup—logging, rebuilding assets, etc. That may be much faster done to a native disk than into a docker container, regardless of whether the volume is shared or not.

1 Like

Yep! This definitely works, and you should stick to whatever is working for you :slight_smile:.

This was a great suggestion! I have this working and will update have updated the instructions above.

1 Like

It might clarify a lot of the confusion here if this is highlighted in the topic: this setup only runs postgres and redis in docker, and Rails itself is ran natively on macOS.

This is the biggest difference vs d/boot_dev, and would explain the performance difference.

I personally don’t think it is a huge deal what images (Bitnami or not) are used for redis and postgres in development, unless it could lead to discrepancies in behaviour…

In short,

11 Likes

Thanks! Tried to clarify that.

A few other thoughts before I let this go:

@pfaffman: one source of confusion is that you seem to be implying that the changes made to the default postgres (and probably redis) configurations by the launcher script are also made during development. I don’t see where that happens—at least not the official native macOS guide. So, while these edits are probably important for production deployments, they don’t seem to be required to set up a working development database and get all of the tests passing.

@sam: the performance of this approach could be further improved by using the native postgres that asdf will install and pin to the project for you, since then its IO is also outside of a container. (I would hope that Redis isn’t super IO bound…) I decided not to do this since it adds an extra command to the startup process, but it’s definitely possible. As noted above, this postgres would not have the deep customization done to prepare the production data container, but that doesn’t seem to be strictly necessary for development. At that point you’d just need to install redis—which sadly asdf won’t do without brew.

1 Like

Oh so this is just about installing pg and redis

I guess I am mega confused why is ‘brew install redis’ and ‘brew install postgres’ considered a challenge to a developer?

4 Likes