Running Discourse on Docker for Mac

[xrav3nz]

docker-sync user here

Essentially, I created a modified version of bin/docker/boot_dev to start the container in detached mode with the docker-sync'ed volumes, so the rest of the Discourse bin/docker scripts would still work.

---

This is how I structured everything.

.
|-- src/
|-- docker-sync.yml
|-- up
|-- down

src/ is the cloned Discourse git repo.

up is the aforementioned modified boot_dev script, to docker-sync start and start the docker container; down stops docker-sync and the docker container.

My `docker-sync.yml`

version: "2"

options:
  verbose: true
syncs:
  discourse-sync:
    src: './src'
    sync_userid: '1000'
    sync_groupid: '1000'
    sync_excludes: ['tmp/cache', '.DS_Store', '._*', 'data/postgres*', 'log/development.log']
  postgres-sync:
    src: './src/data/postgres'
    sync_userid: '107'
    sync_groupid: '112'

Critical snippet in `up`

docker-sync start

docker run -d -p 1080:1080 -p 3000:3000 \
 --volumes-from discourse-sync \
 --volumes-from postgres-sync \
 -v "postgres-sync:/shared/postgres_data:nocopy" \
 -v "discourse-sync:/src:nocopy" \
 -e BUNDLE_PATH=/src/vendor/bundle \
 --hostname=discourse \
 --name=discourse_dev \
 --restart=always \
 discourse/discourse_dev:release /sbin/boot

I kinda forgot why I have two separate sync’ed volumes: one for the database and another for the source code.

Never got around to perfect the script since it just works Lemme know if you’ve any questions!

[DiscourseMetrics]

Thanks! @xrav3nz! That was very helpful

I think I’m very close to getting it to work.

Here’s what I did:

1. Used your docker-sync.yml, put it in the directory containing my app directory.
2. Duplicated the boot_dev script and replaced the docker run -d... line with the content in your second snippet. Saved it as up in the same directory as the docker-sync file.
3. My app directory is named discourse, so I replaced src with discourse in both of the snippets.
4. Ran ./up in the directory containing the up file.
5. Docker-sync and Docker start

How do I start rails? I tested running ./up --init which fails with this error:

Installing gems...
./up: line 47: /Users/jt/app/bundle: No such file or directory

On that line it attempts to run "${SCRIPTPATH}/bundle" install so it is not registering as a regular “bundle install”

If I run ./bin/docker/rails s it now fails too saying that I’m not doing the command correctly:

Usage:
  rails new APP_PATH [options]

I’m sure I’m just missing a small puzzle piece here Thanks again

[xrav3nz]

Since up is not in the bin/docker directory with the rest of the scripts, $SCRIPT_PATH is incorrect and thus $SCRIPT_PATH/bundle does not exist.

The easiest fix would be to only start docker-sync and the docker containers in up, and run the rest of the scripts separately.

For example,

$ ./up

# this is the initialization process in `boot_dev --init`
$ discourse/d/bundle install
$ discourse/d/rake db:migrate
$ discourse/d/rake admin:create

$ discourse/d/rails s

I hope it works this time, let me know how it goes.

[DiscourseMetrics]

Thanks!

Running discourse/d/bundle install in the directory containing discourse says:
Could not locate Gemfile

Running discourse/d/rails s gives the same error as in my previous post.

[DiscourseMetrics]

discourse/d/bundle install --gemfile=/discourse/Gemfile seems to work…

[xrav3nz]

Hmmm interesting…

In up:

docker run -d -p 1080:1080 -p 3000:3000 \
 --volumes-from discourse-sync \
 --volumes-from postgres-sync \
 -v "postgres-sync:/shared/postgres_data:nocopy" \
 -v "discourse-sync:/src:nocopy" \
 -e BUNDLE_PATH=/src/vendor/bundle \
 --hostname=discourse \
 --name=discourse_dev \
 --restart=always \
 discourse/discourse_dev:release /sbin/boot

Did you replace the src in -v "discourse-sync:/src:nocopy" to discourse as well? This src refers to the directory in the container. Sorry if my folder names are a bit confusing…

[DiscourseMetrics]

Yes, I did Is there anything else that has been changed in the up file?

[xrav3nz]

I was trying to say that -v "discourse-sync:/src:nocopy" should not be changed, otherwise the source code would be mounted to a different directory in the container. This explains why neither rails nor bundler could find the source code.

[DiscourseMetrics]

D’oh Brilliant, thank you! Having this up and running makes dev work so much easier!

I owe you a

[sam (Sam Saffron)]

Be sure to do a PR to discourse making this config optional so we both document and make switches for it.

[DiscourseMetrics]

So I had it working for a few minutes. Then I noticed that files weren’t actually syncing after making changes. Figured I’d try a docker-sync clean and do the whole thing again. I do the same steps we just went through, but again, I get the Gemfile not found message. However, this time I haven’t renamed src to discourse. Not sure what is happening here.

I’ve even tried renaming the discourse folder to src to match your code exactly, but it still won’t work.

Is there a way to check if the source code is mounted to the right directory?

[xrav3nz]

Is there a way to check if the source code is mounted to the right directory?

d/shell into the container and check manually? docker-sync on macOS is not quite stable (judging from its GitHub issues), but I’ve rarely had problems myself.

---

I added the docker-sync related functionalities to Discourse and pushed it to a branch on my fork:
xrav3nz:dev/docker-sync. Would you like to try it out instead? docker-sync is integrated into boot_dev and shutdown_dev, and all the existing docker scripts should continue to work.

I’ll PR the changes into core if everything looks good for the next few days.

[DiscourseMetrics]

Nice! Unfortuntely, even on your branch I get “Could not locate Gemfile”

in /discourse: $ ./bin/docker/boot_dev --init
Using source in: /Users/jorgen/Dropbox/Design/2018/discourse
Using data in:   /Users/jorgen/Dropbox/Design/2018/discourse/data/postgres
          ok  Starting native_osx for sync discourse-sync
discourse-sync
unison: stopped
unison: started
     success  Sync container started
          ok  Starting native_osx for sync postgres-sync
postgres-sync
unison: stopped
unison: **ERROR (spawn error)**
     success  Sync container started
     success  Starting Docker-Sync in the background
2fb1ebd82c77cb705b96c10069e5ec5970b8aa13417390114a34461f00237f48
Installing gems...
Could not locate Gemfile

There is a “spawn error” near the end there…not sure if that has anything to do with it.

[xrav3nz]

Could you try d/shutdown_dev && docker-sync clean && rm .docker-sync/daemon.log && d/boot_dev, and PM me the content of .docker-sync/daemon.log? Please do not add the --init option just yet, since that will obfuscate the log.

[DiscourseMetrics]

Ah, I had forgotten to do a docker-sync clean after switching to your code, it seems to work as it should now Thanks again

[cmwebdev (Colin Marshall)]

Sorry but I am having trouble understanding the setup process to get Discourse working locally with Docker for Mac. I’m not that great with Docker yet, so bear with me.

The first thing I did was run:
docker pull discourse/base:release

That downloads fine. Then I create a container by running:
docker run --discourse -d discourse/base:release

What’s the next thing I want to do after that?

I see the instructions here:
https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md

I can’t do docker exec -it discourse /bin/bash because the container doesn’t stay running after I run it.

Thanks for the assistance!

[ChrisBeach (Chris)]

I added the docker-sync related functionalities to Discourse and pushed it to a branch on my fork:

xrav3nz:dev/docker-sync . Would you like to try it out instead? docker-sync is integrated into boot_dev and shutdown_dev, and all the existing docker scripts should continue to work.

Hi @xrav3nz - thanks for doing this work. I find macOS/Docker/Discourse runs very slowly compared to Linux/Docker/Discourse

I tried using your branch but got the following on `d/boot_dev --init`: pg_dump: [archiver] could not open output file "/src/db/structure.sql": Permission denied Any recommendations?

Update - it appears to have worked after retrying the boot_dev command. Unsure what caused the original problem.

[xrav3nz]

TL;DR: the Docker for Mac delegated/cached volume mount is consistently ~3x faster now, and much more stable than docker-sync. I’d be happy to PR a change when I’m sure this does not introduce new problems.

---

I’ve been running into more and more problems with docker-sync lately. Unfortunately, I don’t understand enough of it to figure out what exactly is wrong.

I tried out the different mount options from Sam’s post a year ago,

Running Discourse on Docker for Mac

Following: Doing dev work in docker on a mac may be workable! Rails Boot Slowdown Host 5.64 Virtual Box 6.15 9% Virtual Box + Docker 8.78 55% Docker Mac (cached) 11.2 98% Docker Mac (uncached) 15.7 178% Note: for Docker setup on virtual box has gems are in AUFS and working directory in a mounted volume. On mac the working directory is a cache mounted volume. It is likely performance would improve if overlay was used or gems are mounted on the host to avoid virtualized file s…

• docker-sync

  ❯ /usr/bin/time d/rails r 'puts'
  2.71 real         0.07 user         0.02 sys
  3.11 real         0.07 user         0.02 sys
  2.99 real         0.07 user         0.02 sys
  

• default (consistent) i.e. -v "$SOURCE_DIR:/src"

  ❯ /usr/bin/time d/rails r 'puts'
  13.74 real         0.06 user         0.01 sys
  13.73 real         0.06 user         0.01 sys
  14.44 real         0.06 user         0.01 sys
  

• delegated i.e. -v "$SOURCE_DIR:/src:delegated"

  ❯ /usr/bin/time d/rails r 'puts'
  4.77 real         0.07 user         0.01 sys
  5.39 real         0.06 user         0.07 sys
  4.86 real         0.06 user         0.01 sys
  

I have been using the delegated mount for the past two weeks, and have yet to ran into any problems. Even though docker-sync is still the most performant, delegated/cached is stable and consistent, and it’s already fast enough (~3x faster than the default) for me (running tests and localhost).

[sam (Sam Saffron)]

Sure send through a pr, looks great

[ChrisBeach (Chris)]

TL;DR: the Docker for Mac delegated/cached volume mount is consistently ~3x faster now, and much more stable than docker-sync. I’d be happy to PR a change when I’m sure this does not introduce new problems.

That’s great news!

I tried to get docker-sync working but struggled. Having another option comes as a relief!