Advanced Troubleshooting with Docker

Sometimes, things will break. Because Docker makes things a little bit different than what you may be used to, this document intends to acquaint you with what’s different in Discourse so that you can troubleshoot to the best of your abilities.

Host vs Container

Discourse runs inside of a Docker container, while Docker is running on the host. When you ssh into your server, you are presented with a prompt like this:

kane@newlaptop:~$ ssh
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:

  System information as of Fri May 16 11:01:36 PDT 2014

  System load:  0.07               Processes:              139
  Usage of /:   81.2% of 29.40GB   Users logged in:        0
  Memory usage: 47%                IP address for eth0:
  Swap usage:   0%                 IP address for docker0:

  Graph this data and manage this system at:

Last login: Fri May 16 11:01:36 2014 from ***

This is the host. The host should have:

  • The /var/discourse folder

  • Docker installed

  • unattended-upgrades running

  • Any other services sharing the same machine running. This is your VPS - you can do whatever you want with it.

The container is where Discourse runs. You can get a shell in the container by running ./launcher enter app. When you run that command, you should be presented with a prompt like this:

root@discourse:/var/discourse# ./launcher enter app

Welcome to Discourse Docker

Use: rails, rake or discourse to execute commands in production

Last login: Fri May 16 18:01:43 2014 from

This is the container. The container has the /var/www/discourse folder, it runs nginx and thin, as well as PostgreSQL and Redis (unless you have the multi-container setup).

In general, changes you make to the container will NOT survive a rebuild (a destroy/bootstrap cycle). If you want to make a permanent change to the container, you want to place it into the container definition - your app.yml file.

However, changes to the database will survive a rebuild. The database is actually stored on the host, in the /var/discourse/shared/standalone directory, and symlinked into the container as /shared. This is defined in the volumes section of the container definition.

Example: Rebuild the container

Let’s run through a task demonstrating the difference between the host and the container.

Rebuilding the container should actually be a moderately routine task. Whenever the container definition changes, you will need to rebuild it. Because we’re performing an operating “on” the container - affecting its lifecycle - this is done from the host.
Because it’s done so often, there’s a command in the ./launcher for it.

root@discourse:/var/discourse# git pull
Already up-to-date.
root@discourse:/var/discourse# ./launcher rebuild app

(lots of output....)


That’s all there is to it. Now let’s run through a separation example.

Example: Host/Container separation and sharing

This is an exercise to demonstrate the separation and relationship between the Host and the Container.

Starting from the host, walk through these commands.

root@discourse:/var/discourse# ./launcher enter app
root@forum:~# echo "Some content" > /root/my_file
root@forum:~# echo "Different content" > /shared/shared_file
root@forum:~# cat /root/my_file
Some content
root@forum:~# cat /shared/shared_file
Different content
root@forum:~# logout
root@discourse:/var/discourse# ls shared/standalone
backups          postgres_data      redis_data   uploads
log              postgres_data_old  shared_file  vendor_bundle
postgres_backup  postgres_run       ssl

#                                   ^ see shared_file there?

root@discourse:/var/discourse# cat ./shared/standalone/shared_file
Different content
root@discourse:/var/discourse# ./launcher rebuild app
-- omitted --
root@discourse:/var/discourse# cat ./shared/standalone/shared_file
Different content
root@discourse:/var/discourse# ./launcher enter app
root@forum:~# cat /root/my_file
cat: /root/my_file: No such file or directory
root@forum:~# cat /shared/shared_file
Different content
root@forum:~# logout
root@discourse:/var/discourse# logout


Note how the file in /root was apparently “deleted” by the rebuild, but the file in shared was both not deleted, and visible on the host!

Example: Delete a corrupted database

Sometimes you screw up the database and need to restore from a backup. However, if the broken database is making the site unusable, you may not be able to get to /admin/backups to upload and restore the old backup. When this happens, you will want to nuke the database.

From the host:

root@discourse:~# cd /var/discourse
root@discourse:/var/discourse# ./launcher destroy app
root@discourse:/var/discourse# echo rm -rv shared/standalone/postgres_*
rm -rv shared/standalone/postgres_backup shared/standalone/postgres_data shared/standalone/postgres_data_old shared/standalone/postgres_run
root@discourse:/var/discourse# ./launcher bootstrap app
root@discourse:/var/discourse# ./launcher start app

Example: Install a plugin

To install a plugin in Discourse, it needs to be placed in /var/www/discourse/plugins. However, this is inside the container - and changes to the container are wiped when it is rebuilt! So, the container definition is what needs to be edited.

Remember, the app.yml is a YAML file, so correct spaces are extremely important. If you download the file for editing, try running it through a YAML validator before reuploading the changed file. If you edit the file on the server, keep your finger off the TAB key and line up all your columns.

There’s already a template in the app.yml file for installing plugins due to the docker_manager plugin, so just add your plugin on the end!

    - exec:
        cd: $home/plugins
          - git clone
          - git clone
          - git clone

The Rails console

The Rails console is accessible by running rails c once inside the container. You can use it to query and edit the database directly. Warning: Messing with the database may result in your Discourse installation being broken in strange ways. Make a backup before any manual database operations.

Here, we’ll run through a few example maintenance tasks to illustrate the differences between the container and the host.

Example: Query the database

Let’s find all of the topics that have exactly 2 posts in them. We’ll need to query the database, and this is also something that would be pretty nice to use Rails for.

Let’s enter the container and get a rails console:

root@discourse:/var/discourse# ./launcher enter app
root@forum:~# rails c
2014-05-16T18:23:24Z 303 TID-ovzipu9vw INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
Loading production environment (Rails 4.1.1)
[1] pry(main)> 

The command we want to execute will look something like Topic.where(???).count. We need to figure out how to construct the where query. Let’s try looking at all the fields Topic has:

[2] pry(main)> Topic
=> Topic(id: integer, title: string, last_posted_at: datetime, created_at: datetime, updated_at: datetime, views: integer, posts_count: integer, user_id: integer, last_post_user_id: integer, reply_count: integer, featured_user1_id: integer, featured_user2_id: integer, featured_user3_id: integer, avg_time: integer, deleted_at: datetime, highest_post_number: integer, image_url: string, off_topic_count: integer, like_count: integer, incoming_link_count: integer, bookmark_count: integer, star_count: integer, category_id: integer, visible: boolean, moderator_posts_count: integer, closed: boolean, archived: boolean, bumped_at: datetime, has_summary: boolean, vote_count: integer, archetype: string, featured_user4_id: integer, notify_moderators_count: integer, spam_count: integer, illegal_count: integer, inappropriate_count: integer, pinned_at: datetime, score: float, percent_rank: float, notify_user_count: integer, subtype: string, slug: string, auto_close_at: datetime, auto_close_user_id: integer, auto_close_started_at: datetime, deleted_by_id: integer, participant_count: integer, word_count: integer, excerpt: string, pinned_globally: boolean)

Well, that’s a lot to look through. But start from the top - right on the second line is posts_count: integer! That sounds a lot like what we want. But, just to make sure, let’s check a topic we know about. To get a specific topic, look in the URL for that topic:

And pull out the first number - 24, then run Topic.find(24). Our topic has 5 posts, so we want posts_count to be 5.

[3] pry(main)> Topic.find(24)
=> #<Topic id: 24, title: "This is my example topic", last_posted_at: "2014-05-16 18:28:30", created_at: "2014-04-07 17:16:58", updated_at: "2014-05-16 18:28:32", views: 11, posts_count: 5, user_id: 3, last_post_user_id: 1, reply_count: 2, featured_user1_id: 5, featured_user2_id: nil, featured_user3_id: nil, avg_time: 19, deleted_at: nil, highest_post_number: 5, image_url: nil, off_topic_count: 0, like_count: 0, incoming_link_count: 0, bookmark_count: 0, star_count: 0, category_id: 5, visible: true, moderator_posts_count: 0, closed: false, archived: false, bumped_at: "2014-05-16 18:28:30", has_summary: false, vote_count: 0, archetype: "regular", featured_user4_id: nil, notify_moderators_count: 0, spam_count: 0, illegal_count: 0, inappropriate_count: 0, pinned_at: nil, score: 0.775, percent_rank: 0.419354838709677, notify_user_count: 0, subtype: nil, slug: "this-is-my-example-topic", auto_close_at: nil, auto_close_user_id: nil, auto_close_started_at: nil, deleted_by_id: nil, participant_count: 3, word_count: 29, excerpt: "Foo bar what's up baz", pinned_globally: false>

And, there it is! posts_count is indeed 5, which is what we wanted. Let’s plug that into our query that we wanted to run:

[4] pry(main)> Topic.where(posts_count: 2).count
=> 3

And there we have it! There are 3 topics with 2 posts in them. All done.

You can do some quite complicated queries, and also modify the database with this. If you need intermediate steps, assign your results to a variable, as seen in this example of manually setting an account to be an admin:

[2] pry(main)> u = User.find_by_username("riking")
=> #<User id: 1, username: "riking", .....snip.......>
[3] pry(main)> u.admin = true
=> true
[4] pry(main)>
=> true
[5] pry(main)> exit

However, be careful when modifying the database - it may be possible to make your site unusable, so make a backup in /admin/backups and download it first!


Is the line below still necessary in the new discourse? I keep seeing it on Readme pages of plugins, meanwhile, it isn’t there in the current app.yml file.


No, it is no longer necessary.


Login via SSH not working

./launcher ssh app
Usage: launcher COMMAND CONFIG [–skip-prereqs] [–docker-args STRING]
start: Start/initialize a container
stop: Stop a running container
restart: Restart a container
destroy: Stop and remove a container
enter: Open a shell to run commands inside the container
logs: View the Docker logs for a container
bootstrap: Bootstrap a container for the config based on a template
rebuild: Rebuild a container (destroy old, bootstrap, start new)
cleanup: Remove all containers that have stopped for > 24 hours

–skip-prereqs Don’t check launcher prerequisites
–docker-args Extra arguments to pass when running docker
–skip-mac-address Don’t assign a mac address


That has been removed quite some time back. ./launcher enter app will work fine :slight_smile:


Working :wink:

I add this to app.yml

  • “templates/sshd.template.yml”


  • “2220:22” #sshd

How do I now login via SSH? default root password or something ?

I killed the sshd template, if you need to ssh into the container.


  • Build your own template that runs sshd and configures it to taste.

  • SSH into host and then enter the container (this can easily be chained and is what we do interally)

Im getting this when I try it.

# ./launcher enter app
-bash: ./launcher: No such file or directory

First you need to go to the Discourse folder, like cd /var/discourse


I need to set up a Dscourse on a vps with over 10 websites in it.

this article said:

Where is this guide? I got lost in all the variation of the content on the post. Is there a better guide explaining how to do this? My entire server stopped working following a basic install and all my websites got dropped. Which instructions should I follow to make this work? It seems like its almost impossible.

Maybe just an error in your nginx configuration files? What is the output of nginx -t


How to connect to a secondary site in multisite setup?

I keep forgetting the environment variable to switch host/db in a multisite console…

cd /var/discourse
./launcher enter web
RAILS_DB=db_name rails c

Note: db_name is not actually the database name, but the entry name for the database in multisite.yml. So for example if you have:

    - file:
        path: $home/config/multisite.yml
        contents: |
            adapter: postgresql
            database: discourse1
            username: discourse
            password: "super sekrit"
            host: data
            pool: 25
            timeout: 5000
            db_id: 1
            adapter: postgresql
            database: discourse2
            username: discourse
            password: "super sekrit"
            host: data
            pool: 25
            timeout: 5000
            db_id: 2

You can access the discourse2 database using its handle: secondary.

cd /var/discourse
./launcher enter web
RAILS_DB=secondary rails c

Good tutorial liked it.

I’ve been able to rebuild my app a few times over the past 6 months, but today I run into network error messages like these:

Retrying download gem from due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (

After consulting various sources, I already tried to disable ipv6 by starting the launch command with extra args:

./launcher rebuild app --docker-args "--sysctl net.ipv6.conf.all.disable_ipv6=1 --sysctl net.ipv6.conf.default.disable_ipv6=1"

To no avail.

What’s happening? How to solve this network issue?


1 Like

Rubygems has been known to rate limit.

Thanks for your quick response.
Is there a way to get around this hurdle? Specify another server? First downloading the files needed? How.

When I tried to rebuild the app again, I immediately noticed a fresh download at the start of the process. Whoever produced a fresh docker version, thank you very, very much. The rebuild when smoothly.

I am getting this same error here, and whatever I tried for two days is not seeming to work for me. Is there anyone who could give me a direction to solve this problem or any advice ? I am not even able to execute the “./launcher enter app” command, I think Docker even does not start running, because bootstrapping the installation didn’t finish succesfully. :frowning:

I am frequently getting this same error (Gem::RemoteFetcher::UnknownHostError) here for the past few days. And often I have to rebuild the app 20+ times in order to get it to work! Are there any workaround/tips to get this working again?

I just can’t believe I spent so much time figuring this out!
In my case the gateway interface (eth0) had MTU of 1450 and docker0 had MTU of 1500. You can check it with:
ip addr | grep mtu
So I tried to lower the MTU value for docker0 interface by creating a daemon.json file in /etc/docker:

$ cat /etc/docker/daemon.json
  "mtu": 1400

and restarted the Docker daemon: service docker restart or your way of choice.
All of my builds are going through without any problem since then!

Hope it helps someone in need just like me! :slight_smile: