Issue regarding Docker development image

Dev
1

I’ve tried to setup my Discourse development environment the Docker way, but I’ve realized that boot_env script tries to pull a non existent discourse/discourse_dev:latest docker image. Instead, a discourse/dev:latest image exists in Docker Hub, but it is outdated (version 1.0.13).

So I’ve tried to build it myself running

$ sudo ruby build.rb -i discourse_dev

But when I run

$ ./bin/docker/boot_env --init

I get an error message regarding PostgreSQL:

Migrating database...
URGENT: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
 Failed to initialize site default
URGENT: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
 Failed to initialize site default
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Any ideas to fix it?

2 Likes
2

So the image build successfully for you, good to know.

Let me take a look at this.

1 Like
3

I too am interested in an official well maintained docker image for development where I can simply mount my working directory and map a port then land into the bash prompt of the container. It should ship with some setup scripts to make the engines cracking inside.

Additionally, it would be good to have a developer documentation maintained similar to the one at:

4

I tried the following the Developing using Docker guide:

$ cd /tmp
$ git clone https://github.com/discourse/discourse.git
$ cd discourse
$ ./bin/docker/boot_dev --init

Realized that it is looking for discourse/discourse_dev:latest image which doesn’t exist in the DockerHub repository. So I decided to build it locally and tag accordingly:

$ cd /tmp
$ git clone https://github.com/discourse/discourse_docker.git
$ cd discourse_docker
$ docker build -t discourse/discourse_dev image/discourse_dev

It complained about missing template files and failed to build. So I looked at the corresponding Dockerfile to find out what files are being added then I copied them from the template directory:

$ cp templates/redis.template.yml templates/postgres.template.yml image/discourse_dev
$ docker build -t discourse/discourse_dev image/discourse_dev

Voila! Success! :slight_smile:

Going back to the main discourse directory to setup the work environment:

$ cd /tmp/discourse
$ ./bin/docker/boot_dev --init

Container spun and gems bundled, but database migration failed:

Bundle complete! 98 Gemfile dependencies, 184 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Post-install message from certified:
IMPORTANT: Remember to use the included executable `certifed-update` regularly to keep your certificate bundle up to date.
Migrating database...
Rails Error: Unable to access log file. Please ensure that /src/log/development.log exists and is writable (ie, make it writable for user and group: chmod 0664 /src/log/development.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /src/tmp/ember-rails/ember.js
/src/config/environment.rb:5:in `<top (required)>'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
Creating admin user...
Rails Error: Unable to access log file. Please ensure that /src/log/development.log exists and is writable (ie, make it writable for user and group: chmod 0664 /src/log/development.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /src/tmp/ember-rails/ember.js
/src/config/environment.rb:5:in `<top (required)>'
Tasks: TOP => admin:create => environment
(See full trace by running task with --trace)

So, I decided to dive inside the running container and run the migrations manually:

$ docker exec -it discourse_dev bash
# cd src
# rake db:migrate

But the migrations did not succeed.

URGENT: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
 Failed to initialize site default
URGENT: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
 Failed to initialize site default
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Tasks: TOP => db:migrate
(See full trace by running task with --trace)

From here I am not sure what else should I try.

3 Likes
5

@Falco can you play around with the dev image and see if you can get it going, maybe document it a bit better?

5 Likes
6

For last few days I was looking into the Discourse’s code base and trying to understand it’s working so that I can make my hands dirty with some development and contribution. However, not being able to setup the development environment, I can’t do much. I hope @Falco will be able to resolve it sooner than later.

The commands that I listed in a sequence in my last post should be reproducible.

7

Sorry about that, we found a nasty bug with our memory allocation with lastest docker image and I’ll fix that before getting the dev image out.

I started looking at it tough, and it’s mostly done, we just need to get PostgreSQL to work.

5 Likes
8

While it’s great that you’re testing the new Docker based workflow, keep in mind that this isn’t our default development environment yet. Our beginners guide for Ubuntu and Mac still work fine.

2 Likes
9

Meanwhile, I have setup a fullly functional and standard Docker development environment, dockerizing the set of instructions in Discourse Advanced Developer Install Guide.

Check it out here: https://github.com/adab1ts/discourse/commit/f2ff9aecf31794d032349646a3dfc8bd94387363

Changes in development workflow:

Development

First time:

$ docker-compose build
$ docker-compose run discourse bundle exec rake db:create db:migrate
$ docker-compose run discourse bundle exec rake admin:create
$ docker-compose up -d
$ docker-compose exec discourse mailcatcher --ip=0.0.0.0
$ docker-compose exec discourse bundle exec sidekiq -q critical,low,default -d -l log/sidekiq.log
$ docker-compose stop

Common workflow:

$ docker-compose start
$ docker-compose exec discourse mailcatcher --ip=0.0.0.0
$ docker-compose exec discourse bundle exec sidekiq -q critical,low,default -d -l log/sidekiq.log
$ docker-compose stop

Test

$ docker-compose run -e "RAILS_ENV=test" discourse bundle exec rake db:create db:migrate
$ docker-compose run discourse bundle exec rake autospec p l=5

To link services successfully you have to reference their names as host values in config/database.yml and config/discourse.{development,test}.conf:

# config/discourse.development.conf

# comma delimited list of emails that have developer level access
developer_emails = carles@adabits.org

# redis server address
redis_host = redis

# redis server port
redis_port = 6379

# redis slave server address
redis_slave_host =

# redis slave server port
redis_slave_port = 6379

# redis database
redis_db = 0

# redis password
redis_password =

# redis sentinels eg
# redis_sentinels = 10.0.0.1:26381,10.0.0.2:26381
redis_sentinels =
5 Likes
10

FROM buildpack-deps:xenial

not using our base image loses out tons of the advantages of using docker in dev. We are super careful to include all the fiddly dependencies in our docker image including image magick, nginx and plenty of other bits and pieces.

compose is fine if that is your thing but you should base of our base image or discourse/discourse image.

1 Like
11

This sort of works, but there are some issues that need careful consideration. Especially the fact the UIDs inside docker do not necessarily the UIDs outside of docker, it is a delicate game that you need to play.

12

I wrote that before I realized that there is a docker image for development that has some workflow in place already.

13

It is just a temporary solution to face the issues reported in my first comment. I will adopt the official docker workflow as soon as it is fixed. In case you weren’t aware of, notice that bin folder is not being tracked by git (https://github.com/discourse/discourse/blob/master/.gitignore#L11) so any changes to bin/docker/boot_dev are being ignored.

discourse/discourse image is not for development as it excludes the gems in development group of Gemfile (https://github.com/adab1ts/discourse_docker/blob/master/image/discourse/Dockerfile#L17).

discourse/base is a bit time consuming to reason about, that’s why I followed the steps in your easy to follow Discourse Advanced Developer Install Guide.

But again, I’m not interested in my own way. As I said before, I will adopt the official docker workflow as soon as it is fixed.

3 Likes
14

I spent some time cleaning stuff up today and have this all working.

Follow the instructions here and you should be good:

https://github.com/discourse/discourse/blob/master/bin/docker/README.md

(if you tried this in the past try running sudo rm -fr tmp/postgres and docker pull discourse/discourse_dev to clean up stuff)

Let me know how you go.

5 Likes
15

It looks like the said image is still not in the DockerHub. Ideally it should be configured as an auto-build there.

$ ./bin/docker/rails s
Error response from daemon: No such container: discourse_dev
16

Have you tried readme, step by step?

17

My bad, I missed the init step. I thought I did run that.

18

Worked smoothly like some fresh butter. @sam you are awesome!

On a side note, I read somewhere that it is good idea to run sidekiq and mailcatcher even in the dev mode. Should it be added to the Docker workflow as well?

1 Like
19

Good point, in general I simply disable queue jobs site setting in dev and avoid running sidekiq but you are right should be added, so I went ahead and added both :slight_smile:

Also, to clean up usage I added a symlink to bin/docker called d

So:

d/mailcatcher
d/sidekiq
d/bundle install
... and so on work ..

https://github.com/discourse/discourse/commit/97e2a614c8587bbd9535ed28dc1bcf766751b2cc

5 Likes
20

This is awesome @sam. I am little curious about a command there though.

CMD="mailcatcher --http-ip 0.0.0.0 -f || (apt-get install -y libsqlite3-dev && gem install mailcatcher && mailcatcher --http-ip 0.0.0.0 -f)"

Why do we have a fallback (or ensuring) installation command here? Why can’t it be made sure in the Docker image beforehand?

1 Like