Beginners Guide to Install Discourse for Development using Docker

dev-install

(Rishabh Nambiar) #1

Developing using Docker

Since Discourse runs in Docker, you should be able to run Discourse directly from your source directory using a Discourse development container.

:white_check_mark: Pros: No need to install any system dependencies, no configuration needed at all for setting up a development environment quickly.

:x: Cons: Will be slightly slower than the native dev environment on Ubuntu, and much slower than a native install on MacOS.

Step 1: Install Docker

Ubuntu

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce

MacOS

Option 1: Download a packaged .dmg from the Docker store
Option 2: brew install docker

Step 2: Start Container

(from your source root)

./bin/docker/boot_dev --init
    # wait while:
    #   - dependencies are installed,
    #   - the database is migrated, and
    #   - an admin user is created (you'll need to interact with this)

./bin/docker/unicorn
    # start rails server with sidekiq

or 


./bin/docker/unicorn -x
    # start rails server without sidekiq

… then open a browser on http://localhost:9292 and voila!, you should see Discourse.


Notes:

  • When you’re done, you can choose to kill the Docker container with:

    ./bin/docker/shutdown_dev
    
  • Data is persisted between invocations of the container in your source root tmp/postgres directory.
    If for any reason you want to reset your database run

    sudo rm -fr tmp/postgres
    
  • If you see errors like “permission denied while trying to connect to Docker”, Run:

    run `sudo usermod -aG docker ${USER}` 
    sudo service docker restart
    
  • The Dockerfile comes from discourse/discourse_docker on GitHub, in particular image/discourse_dev.

Running Tests

bin/rake autospec
or
d/bundle exec rspec spec/.../..._spec.rb

HOWTO Set up a development environment using vagrant (Ubuntu guide)
Beginners Guide to Install Discourse on Ubuntu for Development
IDE and comfortable DEV environment for newbies
Using a launcher built docker image in docker-compose
Can't set up dev environment due to cppjieba_rb failing to install
Using a launcher built docker image in docker-compose
User API keys specification
Docker install on MacOS
User API keys specification
OAuth2 and Microsoft ADFS
Using a launcher built docker image in docker-compose
(codechef) #2

Great post @rishabhn. Love the simplicity of the Docker setup. Looking forward to migrating away from Vagrant.

I’m on day two of troubleshooting this and thought I’d see if you’ve run into the issue? On a fresh docker dev setup, I get the following error when I try to backup (or restore.)

pg_dump: [archiver (db)] connection to database "discourse_development" failed: FATAL:  Peer authentication failed for user "postgres"

Repro Steps

  1. On Ubuntu host w/ Docker 18.09.0 (I don’t think this is the issue)
  2. git clone discourse repo
  3. cd ./discourse
  4. ./bin/docker/boot_dev --init
    a. Everything succeeds
    b. I create my admin account
  5. ./bin/docker/rails s
  6. ./bin/docker/sidekiq
  7. ./bin/docker/mailcatcher
  8. Open http://localhost:3000 in chrome
  9. Log in as my admin user
  10. Navigate to the Backup tab and click Backup
  11. Backup fails with the error message above

It seems like a psql security issue, but I’m having a hard time diagnosing the configuration. Any thoughts on how to resolve?

Full Backup Dump

[2018-12-20 16:29:25] 'chad' has started the backup!
[2018-12-20 16:29:25] Marking backup as running...
[2018-12-20 16:29:25] Making sure '/src/tmp/backups/default/2018-12-20-162925' exists...
[2018-12-20 16:29:25] Making sure '/src/public/backups/default' exists...
[2018-12-20 16:29:25] Pausing sidekiq...
[2018-12-20 16:29:25] Waiting for sidekiq to finish running jobs...
[2018-12-20 16:29:25] Dumping the public schema of the database...
[2018-12-20 16:29:25] pg_dump: [archiver (db)] connection to database "discourse_development" failed: FATAL:  Peer authentication failed for user "postgres"
[2018-12-20 16:29:25] EXCEPTION: pg_dump failed
[2018-12-20 16:29:25] /src/lib/backup_restore/backuper.rb:173:in `dump_public_schema'
/src/lib/backup_restore/backuper.rb:38:in `run'
/src/lib/backup_restore/backup_restore.rb:167:in `block in start!'
/src/lib/backup_restore/backup_restore.rb:164:in `fork'
/src/lib/backup_restore/backup_restore.rb:164:in `start!'
/src/lib/backup_restore/backup_restore.rb:18:in `backup!'
/src/app/controllers/admin/backups_controller.rb:38:in `create'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/base.rb:194:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/notifications.rb:168:in `instrument'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.2/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/abstract_controller/base.rb:134:in `process'
/usr/local/lib/ruby/gems/2.5.0/gems/actionview-5.2.2/lib/action_view/rendering.rb:32:in `process'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-mini-profiler-1.0.1/lib/mini_profiler/profiling_methods.rb:104:in `block in profile_method'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal.rb:191:in `dispatch'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_controller/metal.rb:252:in `dispatch'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `each'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/journey/router.rb:35:in `serve'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/routing/route_set.rb:840:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'
/src/lib/middleware/omniauth_bypass_middleware.rb:32:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/src/lib/content_security_policy/middleware.rb:12:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/activerecord-5.2.2/lib/active_record/migration.rb:559:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/better_errors-2.4.0/lib/better_errors/middleware.rb:59:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/logster-1.3.1/lib/logster/middleware/reporter.rb:31:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.2/lib/rails/rack/logger.rb:38:in `call_app'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.2/lib/rails/rack/logger.rb:28:in `call'
/src/config/initializers/100-quiet_logger.rb:16:in `call'
/src/config/initializers/100-silence_logger.rb:29:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.2/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/src/lib/middleware/missing_avatars.rb:21:in `call'
/src/lib/middleware/turbo_dev.rb:34:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-mini-profiler-1.0.1/lib/mini_profiler/profiler.rb:281:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/message_bus-2.2.0.pre.1/lib/message_bus/rack/middleware.rb:57:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.2/lib/rails/engine.rb:524:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `public_send'
/usr/local/lib/ruby/gems/2.5.0/gems/railties-5.2.2/lib/rails/railtie.rb:190:in `method_missing'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/puma-3.11.4/lib/puma/configuration.rb:225:in `call'
/usr/local/lib/ruby/gems/2.5.0/gems/puma-3.11.4/lib/puma/server.rb:632:in `handle_request'
/usr/local/lib/ruby/gems/2.5.0/gems/puma-3.11.4/lib/puma/server.rb:446:in `process_client'
/usr/local/lib/ruby/gems/2.5.0/gems/puma-3.11.4/lib/puma/server.rb:306:in `block in run'
/usr/local/lib/ruby/gems/2.5.0/gems/puma-3.11.4/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
[2018-12-20 16:29:25] Cleaning stuff up...
[2018-12-20 16:29:25] Removing '.tar' leftovers...
[2018-12-20 16:29:25] Unpausing sidekiq...
[2018-12-20 16:29:25] Marking backup as finished...
[2018-12-20 16:29:25] Refreshing disk stats...
[2018-12-20 16:29:25] Notifying 'chad' of the end of the backup...
[2018-12-20 16:29:27] Finished!

Using a launcher built docker image in docker-compose
(Sam Saffron) #3

I guess this is it… my guess is that it should be connecting as the user discourse so something is not set up right. (not your fault likely an internal config issue in the docker image)


(codechef) #4

I tracked it this far, but wasn’t able to make the changes to pg_hba.conf stick, or even test the changes because I wasn’t able to restart the postgresql service.

Note: I found a pg_hba.conf file on my host in ./data/postgres/ though the docker container doesn’t seem to be using that .conf file.


(codechef) #5

@sam - not sure it’s needed on both or even a good solution, but I was able to work around this issue by adding USER=discourse to the ./bin/docker/rails and ./bin/docker/sidekiq bash commands.

Leaving this here incase anyone else runs into it.

New rails command

CMD="cd /src && RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 LD_PRELOAD=/usr/lib/libjemalloc.so RACK_HANDLER=puma RAILS_ENV=${RAILS_ENV:=development} USER=discourse rails $PARAMS"

docker exec -it -u discourse:discourse discourse_dev /bin/bash -c “$CMD”

New sidekiq comand

CMD="cd /src && RAILS_ENV=${RAILS_ENV:=development} USER=discourse bundle exec sidekiq -q critical -q low -q default"

(Sam Saffron) #6

Sure that change seems fine, did a small update here:

Small related recommendation use unicorn which will auto spawn sidekiq instead of running sidekiq standalone. (you will need to nuke the container and boot_dev again to get the port exposed)


(Sam Nazarko) #7

I’m trying to get this up and running but experience some issues when running ./bin/docker/boot_dev -init.

Migrating database...
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
No connection to db, unable to retrieve site settings! (normal when running db:create)
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/src/lib/migration/safe_migrate.rb:50:in `migrate'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

I’ve ensured that tmp/ is writeable; I’ve made sure my user is part of the docker group and I am not running this with privileges, i.e. sudo.

I’ve tried destroying the container with shutdown_dev and starting again after removing the contents of tmp/; but this doesn’t help

Cheers

Sam


(Keith John Hutchison) #9

I logged into my local running instance to enable restoring via discourse enable_restore

I created /usr/local/bin/discourse with the following contents

#!/bin/bash
(cd /src && RAILS_ENV=development sudo -H -E -u discourse bundle exec script/discourse "$@")

#root@discourse:/# discourse enable_restore

Restore are now permitted. Disable them with `disable_restore`

What file(s) would I change to get /usr/local/bin/discourse in a pull request to change the discourse_dev image?


(Sam Saffron) #10

Complete per:


(Alex Armstrong) #11

I want to install Discourse on MacOS using Docker to replace my rickety Vagrant version.

I ran this:

brew install docker
git clone https://github.com/discourse/discourse_docker.git ~/Desktop/discourse
cd ~/Desktop/discourse

Then what? I don’t seem to be able to find boot_dev, etc.


(Alex Armstrong) #13

You linked to the OP, which I’ve read and that is why I’m asking a question in this topic. I’ve done Step 1, but there’s something I need to do between that and Step 2 which isn’t clear to me.


(Keith John Hutchison) #14

Sorry @alehandrof for losing context.

You’ve cloned discourse_docker.git instead of discourse.git.

Try

git clone  https://github.com/discourse/discourse.git

Cheers

Keith


(SteveM) #15

Ubuntu 16.04
I’m getting:

/usr/local/lib/ruby/2.5.0/fileutils.rb:232:in `mkdir': Permission denied @ dir_s_mkdir - /src/tmp (Errno::EACCES)

I’ve tried:

sudo usermod -aG docker ${USER}
sudo service docker restart

But no luck…


(SteveM) #16

Does anyone have a solution for this? I am up a creek without a paddle. I’ve already tried the ubuntu install, but ran into trouble with that. I’m hoping to make this docker install work.


(Kim Miller) #17

If one has a working Discourse instance on DigitalOcean (which I believe was created like so, per community instructions) and wants to experiment / develop with that setup locally on a Mac, how would one go about pulling that down to a local Mac / Docker container?

Or should I just make a fresh, local Discourse Docker install on my Mac and restore a backup copy from my live instance?

I’ve already gotten a normal local Rails instance going on my Mac, but so many of the examples I see show things being done inside a Docker container (e.g. ./discourse-setup) I’ve gotten to wondering which method makes more sense.

Suggestions?


(Sam Saffron) #18

Backup the production instance, then restore it onto a clean development environment using the guide in the OP.


(Kim Miller) #19

… meaning I assume …

Thanks :slight_smile:


(SteveM) #20

I’ve been monkeying around with this a lot. Can anyone tell me why I’m getting permission denied? I’m running as root and I’ve tried with sudo.

I’m not sure if it matters, but I installed docker through plesk, not the command line as shown above.

Bundle complete! 112 Gemfile dependencies, 211 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Migrating database...
rake aborted!
Errno::EACCES: Permission denied @ dir_s_mkdir - tmp
/src/config/boot.rb:21:in `<top (required)>'
/src/config/application.rb:16:in `require'
/src/config/application.rb:16:in `<top (required)>'
/src/Rakefile:5:in `require'
/src/Rakefile:5:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)

Beginner’s Guide…:rofl:


(Justin DiRose) #21

I’ve been getting the same issue with not being able to access Postgres as @samnazarko on a fresh install of Ubuntu. Initially I had an issue with the tmp directory not having permissions, but a simple chmod took care of that.

Any tips for dealing with this? I haven’t been able to get my mind around what the actual problem is to dig further myself.


(Benjamin Raaymakers) #22

Hi guys, I’m wondering why you don’t use docker volume for the postgres data, most of the time performance is much better on Mac and you avoid os specific issues and data is persisted
could be nice to have the option to to pass on the boot_dev to use docker volume instead of local file sharing
DATA_VOLUME=“postgresDockerVolume”
-v “$DATA_VOLUME:/shared/postgres_data:delegated”