Advanced Troubleshooting with Docker


(Kane York) #1

Sometimes, things will break. Because Docker makes things a little bit different than what you may be used to, this document intends to acquaint you with what’s different in Discourse so that you can troubleshoot to the best of your abilities.

Host vs Container

Discourse runs inside of a Docker container, while Docker is running on the host. When you ssh into your server, you are presented with a prompt like this:

kane@newlaptop:~$ ssh root@forum.riking.org
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Fri May 16 11:01:36 PDT 2014

  System load:  0.07               Processes:              139
  Usage of /:   81.2% of 29.40GB   Users logged in:        0
  Memory usage: 47%                IP address for eth0:    107.170.242.76
  Swap usage:   0%                 IP address for docker0: 172.17.42.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

Last login: Fri May 16 11:01:36 2014 from ***
root@discourse:~# 

This is the host. The host should have:

  • The /var/discourse folder

  • Docker installed

  • unattended-upgrades running

  • Any other services sharing the same machine running. This is your VPS - you can do whatever you want with it.

The container is where Discourse runs. You can get a shell in the container by running ./launcher enter app. When you run that command, you should be presented with a prompt like this:

root@discourse:/var/discourse# ./launcher enter app

Welcome to Discourse Docker


Use: rails, rake or discourse to execute commands in production

Last login: Fri May 16 18:01:43 2014 from 172.17.42.1
root@forum:~# 

This is the container. The container has the /var/www/discourse folder, it runs nginx and thin, as well as PostgreSQL and Redis (unless you have the multi-container setup).

In general, changes you make to the container will NOT survive a rebuild (a destroy/bootstrap cycle). If you want to make a permanent change to the container, you want to place it into the container definition - your app.yml file.

However, changes to the database will survive a rebuild. The database is actually stored on the host, in the /var/discourse/shared/standalone directory, and symlinked into the container as /shared. This is defined in the volumes section of the container definition.

Example: Rebuild the container

Let’s run through a task demonstrating the difference between the host and the container.

Rebuilding the container should actually be a moderately routine task. Whenever the container definition changes, you will need to rebuild it. Because we’re performing an operating “on” the container - affecting its lifecycle - this is done from the host.
Because it’s done so often, there’s a command in the ./launcher for it.

root@discourse:/var/discourse# git pull
Already up-to-date.
root@discourse:/var/discourse# ./launcher rebuild app

(lots of output....)

root@discourse:/var/discourse# 

That’s all there is to it. Now let’s run through a separation example.

Example: Host/Container separation and sharing

This is an exercise to demonstrate the separation and relationship between the Host and the Container.

Starting from the host, walk through these commands.

root@discourse:/var/discourse# ./launcher enter app
root@forum:~# echo "Some content" > /root/my_file
root@forum:~# echo "Different content" > /shared/shared_file
root@forum:~# cat /root/my_file
Some content
root@forum:~# cat /shared/shared_file
Different content
root@forum:~# logout
logout
root@discourse:/var/discourse# ls shared/standalone
backups          postgres_data      redis_data   uploads
log              postgres_data_old  shared_file  vendor_bundle
postgres_backup  postgres_run       ssl

#                                   ^ see shared_file there?

root@discourse:/var/discourse# cat ./shared/standalone/shared_file
Different content
root@discourse:/var/discourse# ./launcher rebuild app
-- omitted --
root@discourse:/var/discourse# cat ./shared/standalone/shared_file
Different content
root@discourse:/var/discourse# ./launcher enter app
root@forum:~# cat /root/my_file
cat: /root/my_file: No such file or directory
root@forum:~# cat /shared/shared_file
Different content
root@forum:~# logout
logout
root@discourse:/var/discourse# logout
logout
kane@new-laptop:~$

 

Note how the file in /root was apparently “deleted” by the rebuild, but the file in shared was both not deleted, and visible on the host!

Example: Delete a corrupted database

Sometimes you screw up the database and need to restore from a backup. However, if the broken database is making the site unusable, you may not be able to get to /admin/backups to upload and restore the old backup. When this happens, you will want to nuke the database.

From the host:

root@discourse:~# cd /var/discourse
root@discourse:/var/discourse# ./launcher destroy app
root@discourse:/var/discourse# echo rm -rv shared/standalone/postgres_*
rm -rv shared/standalone/postgres_backup shared/standalone/postgres_data shared/standalone/postgres_data_old shared/standalone/postgres_run
root@discourse:/var/discourse# ./launcher bootstrap app
root@discourse:/var/discourse# ./launcher start app

Example: Install a plugin

To install a plugin in Discourse, it needs to be placed in /var/www/discourse/plugins. However, this is inside the container - and changes to the container are wiped when it is rebuilt! So, the container definition is what needs to be edited.

Remember, the app.yml is a YAML file, so correct spaces are extremely important. If you download the file for editing, try running it through a YAML validator before reuploading the changed file. If you edit the file on the server, keep your finger off the TAB key and line up all your columns.

There’s already a template in the app.yml file for installing plugins due to the docker_manager plugin, so just add your plugin on the end!

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-spoiler-alert.git
          - git clone https://riking-discourse:insecurepassword@bitbucket.org/riking/some-private-plugin.git

The Rails console

The Rails console is accessible by running rails c once inside the container. You can use it to query and edit the database directly. Warning: Messing with the database may result in your Discourse installation being broken in strange ways. Make a backup before any manual database operations.

Here, we’ll run through a few example maintenance tasks to illustrate the differences between the container and the host.

Example: Query the database

Let’s find all of the topics that have exactly 2 posts in them. We’ll need to query the database, and this is also something that would be pretty nice to use Rails for.

Let’s enter the container and get a rails console:

root@discourse:/var/discourse# ./launcher enter app
root@forum:~# rails c
2014-05-16T18:23:24Z 303 TID-ovzipu9vw INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
Loading production environment (Rails 4.1.1)
[1] pry(main)> 

The command we want to execute will look something like Topic.where(???).count. We need to figure out how to construct the where query. Let’s try looking at all the fields Topic has:

[2] pry(main)> Topic
=> Topic(id: integer, title: string, last_posted_at: datetime, created_at: datetime, updated_at: datetime, views: integer, posts_count: integer, user_id: integer, last_post_user_id: integer, reply_count: integer, featured_user1_id: integer, featured_user2_id: integer, featured_user3_id: integer, avg_time: integer, deleted_at: datetime, highest_post_number: integer, image_url: string, off_topic_count: integer, like_count: integer, incoming_link_count: integer, bookmark_count: integer, star_count: integer, category_id: integer, visible: boolean, moderator_posts_count: integer, closed: boolean, archived: boolean, bumped_at: datetime, has_summary: boolean, vote_count: integer, archetype: string, featured_user4_id: integer, notify_moderators_count: integer, spam_count: integer, illegal_count: integer, inappropriate_count: integer, pinned_at: datetime, score: float, percent_rank: float, notify_user_count: integer, subtype: string, slug: string, auto_close_at: datetime, auto_close_user_id: integer, auto_close_started_at: datetime, deleted_by_id: integer, participant_count: integer, word_count: integer, excerpt: string, pinned_globally: boolean)

Well, that’s a lot to look through. But start from the top - right on the second line is posts_count: integer! That sounds a lot like what we want. But, just to make sure, let’s check a topic we know about. To get a specific topic, look in the URL for that topic:

https://forum.riking.org/t/this-is-my-example-topic/24/5

And pull out the first number - 24, then run Topic.find(24). Our topic has 5 posts, so we want posts_count to be 5.

[3] pry(main)> Topic.find(24)
=> #<Topic id: 24, title: "This is my example topic", last_posted_at: "2014-05-16 18:28:30", created_at: "2014-04-07 17:16:58", updated_at: "2014-05-16 18:28:32", views: 11, posts_count: 5, user_id: 3, last_post_user_id: 1, reply_count: 2, featured_user1_id: 5, featured_user2_id: nil, featured_user3_id: nil, avg_time: 19, deleted_at: nil, highest_post_number: 5, image_url: nil, off_topic_count: 0, like_count: 0, incoming_link_count: 0, bookmark_count: 0, star_count: 0, category_id: 5, visible: true, moderator_posts_count: 0, closed: false, archived: false, bumped_at: "2014-05-16 18:28:30", has_summary: false, vote_count: 0, archetype: "regular", featured_user4_id: nil, notify_moderators_count: 0, spam_count: 0, illegal_count: 0, inappropriate_count: 0, pinned_at: nil, score: 0.775, percent_rank: 0.419354838709677, notify_user_count: 0, subtype: nil, slug: "this-is-my-example-topic", auto_close_at: nil, auto_close_user_id: nil, auto_close_started_at: nil, deleted_by_id: nil, participant_count: 3, word_count: 29, excerpt: "Foo bar what's up baz", pinned_globally: false>

And, there it is! posts_count is indeed 5, which is what we wanted. Let’s plug that into our query that we wanted to run:

[4] pry(main)> Topic.where(posts_count: 2).count
=> 3

And there we have it! There are 3 topics with 2 posts in them. All done.

You can do some quite complicated queries, and also modify the database with this. If you need intermediate steps, assign your results to a variable, as seen in this example of manually setting an account to be an admin:

[2] pry(main)> u = User.find_by_username("riking")
=> #<User id: 1, username: "riking", .....snip.......>
[3] pry(main)> u.admin = true
=> true
[4] pry(main)> u.save
=> true
[5] pry(main)> exit

However, be careful when modifying the database - it may be possible to make your site unusable, so make a backup in /admin/backups and download it first!


"How to tinker your discourse" advice for beginners using the Docker install?
Troubleshooting Docker Installation Issues
Multisite configuration with Docker
Destroying docker container does not destroy the database?
Cannot revoke Admin from user
Recommend method of installation?
Using Discourse With Other Sites on Same Droplet
Running other websites on the same machine as Discourse
Can't Increase the Max File Size
Error while pulling translations from Transifex
Getting Discourse installed on OVH VPS
How do I see the files in a Discourse install?
Discourse Solved (Accepted answer plugin)
Where to modify Signup call-to-action content text
Benefits of running discourse in separate containers for data and web
Failed to bootstrap, also 500 error loading any page
Error Trying to Start Import Process - Xenforo to Discourse
Cannot pass REST API queries to the docker container
Starting a second Discourse forum on the same VPS
Is it necessary to install nginx in front of docker?
Activate user on a non-configurated-email-server
How do I use spoiler tags on my own discourse instance
Avatar upload issue
Config Nginx to receive real IP when using Cloudflare for noobs
How do I disable RC4 SSL support?
Turkish translation missing many singular strings
Is there a way to resend the "Welcome to Discourse" intro PM?
(Ndianabasi Udonkang) #11

Is the line below still necessary in the new discourse? I keep seeing it on Readme pages of plugins, meanwhile, it isn’t there in the current app.yml file.


(Kane York) #12

No, it is no longer necessary.


(AndrzejS) #13

Login via SSH not working

./launcher ssh app
Usage: launcher COMMAND CONFIG [–skip-prereqs] [–docker-args STRING]
Commands:
start: Start/initialize a container
stop: Stop a running container
restart: Restart a container
destroy: Stop and remove a container
enter: Open a shell to run commands inside the container
logs: View the Docker logs for a container
bootstrap: Bootstrap a container for the config based on a template
rebuild: Rebuild a container (destroy old, bootstrap, start new)
cleanup: Remove all containers that have stopped for > 24 hours

Options:
–skip-prereqs Don’t check launcher prerequisites
–docker-args Extra arguments to pass when running docker
–skip-mac-address Don’t assign a mac address


(Alan Tan) #14

That has been removed quite some time back. ./launcher enter app will work fine :slight_smile:


(AndrzejS) #15

Working :wink:

I add this to app.yml

  • “templates/sshd.template.yml”

expose:

  • “2220:22” #sshd

How do I now login via SSH? default root password or something ?


(Sam Saffron) #16

I killed the sshd template, if you need to ssh into the container.

Either

  • Build your own template that runs sshd and configures it to taste.

  • SSH into host and then enter the container (this can easily be chained and is what we do interally)


(Roland) #17

Im getting this when I try it.

# ./launcher enter app
-bash: ./launcher: No such file or directory

(Rafael dos Santos Silva) #18

First you need to go to the Discourse folder, like cd /var/discourse


(Hugo Roger) #19

I need to set up a Dscourse on a vps with over 10 websites in it.

this article said:

Where is this guide? I got lost in all the variation of the content on the post. Is there a better guide explaining how to do this? My entire server stopped working following a basic install and all my websites got dropped. Which instructions should I follow to make this work? It seems like its almost impossible.


(Lutz Biermann) #20

Maybe just an error in your nginx configuration files? What is the output of nginx -t


#21

How to connect to a secondary site in multisite setup?

I keep forgetting the environment variable to switch host/db in a multisite console…

cd /var/discourse
./launcher enter web
RAILS_DB=db_name rails c

Note: db_name is not actually the database name, but the entry name for the database in multisite.yml. So for example if you have:

  before_bundle_exec:
    - file:
        path: $home/config/multisite.yml
        contents: |
          main:
            adapter: postgresql
            database: discourse1
            username: discourse
            password: "super sekrit"
            host: data
            pool: 25
            timeout: 5000
            db_id: 1
            host_names:
              - main.example.org
          secondary:
            adapter: postgresql
            database: discourse2
            username: discourse
            password: "super sekrit"
            host: data
            pool: 25
            timeout: 5000
            db_id: 2
            host_names:
              - secondary.example.net

You can access the discourse2 database using its handle: secondary.

cd /var/discourse
./launcher enter web
RAILS_DB=secondary rails c