Configure Discourse - Environment AWS Linux 2 AMI and Apache2 (httpd)

Hi All

I have been through the docs and read through the configuration setups and recommendations posted and have tried to modify the same for my unique situation.

For something that appears to be quite simple, it is incredibly frustrating that none of the proposed solutions have worked.

Our environment:

  • We run an AWS EC2 instance, more specifically an AWS Linux 2 AMI;
  • We have apache2 running and configured;
  • The instance is part of an AWS target group which is referenced by our AWS load balancer;
  • Our SSL / TLS certificate is AWS provided and is associated with the load balancer;
  • Our domain provider is name.com;
  • As part of our setup we have multiple sub domains routed using CNAME records all pointing to the AWS load balancer which then manages traffic to the EC2 instance;
  • All traffic routed to the load balancer and served to the web user is secure;
  • On our server we run three separate node express applications;
  • The ports are defined in express as part of the express server setup, currently ports 3000, 4000 and 5000;
  • Traffic is routed by apache using the sub domain to the relevant web app, for example:
<VirtualHost *:80>
    ServerName subdomain.domain.io
    ProxyPreserveHost On
    ProxyPass "/" "http://localhost:4000/"
    ProxyPassReverse "/" "http://localhost:4000/"
</VirtualHost>

Current status:

  • Docker is running
  • The discourse app (container) is running
  • The express apps are all running and are being properly routed by apache

Notes:

  • I tried running building the app initially without exposing the ports per the documentation, when that didn’t work I did expose port 8000 over http port 80;
  • In my simple understanding docker is like a house and the various containers are the rooms. Docker determines how the app is accessed i.e. which room to access. My thought by exposing port 8000 was that I could then reference port 8000 as I do with my other express apps now that 8000 was exposed. But this does not work, below is an example of what I tried to do in apache:
# FAQ (DISCOURSE ROUTES)

<VirtualHost *:80>
    ServerName discourse.domain.io
    ProxyPreserveHost On
    ProxyPass "/" "http://localhost:8000/"
    ProxyPassReverse "/" "http://localhost:8000/"
</VirtualHost>
  • The simple test was to try and put localhost:8000 into the browser on the server and test. The server is the AWS guid with chromium installed. The fact that this does not resolve immediately tells me that there is an issue and that my understanding is missing something fundamental. My web apps run and are available on the ports I specify in the express setup, my expectation was that it was the same for the docker container.

Additional notes:

  • I realise that there is a potential solution by putting NGINX “in front of” apache and docker to route traffic from discourse.domain.io to say port 8000 on the local server i.e. the port that is exposed for the docker container and also route all other traffic from *.domain.io to apache on say port 9000 and then change my virtual host config to route traffic from port 9000 accordingly, i.e. :
# EXPRESS APP ROUTING IN APACHE

<VirtualHost *:9000>
    ServerName other_sub_domains.domain.io
    ProxyPreserveHost On
    ProxyPass "/" "http://localhost:4000/"
    ProxyPassReverse "/" "http://localhost:4000/"
</VirtualHost>
  • To me this seems completely unnecessary. If the domain discourse.domain.io is routed to apache and apache receives the request and then tries to route it should be as simple as pointing it to the docker container exposed port.
  • Also if localhost:8000 on the server itself isn’t resolving then this would not work anyway.

Currently I get a 502 error in chrome when I try to navigate to my discourse docker container. (See above screenshot).

Extract from app.yml

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/web.socketed.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "8000:80"   # http
  #- "443:443" # https

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  #db_shared_buffers: "256MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  #UNICORN_WORKERS: 3

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: 'discourse.domain.io'

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'user@domain.io'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: email-smtp.us-east-1.amazonaws.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: ******
  DISCOURSE_SMTP_PASSWORD: ******
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)
  #DISCOURSE_SMTP_DOMAIN: ******    # (required by some providers)
  #DISCOURSE_NOTIFICATION_EMAIL: ******  # (address to send notifications from)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com
  
  ## The maxmind geolocation IP address key for IP address lookup
  ## see https://meta.discourse.org/t/-/137387/23 for details
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"

My domain, email address and email config are correct. As mentioned, I do not require SSL/TLS configuration.

Request:

  • Is the NGINX solution the only way I can get this setup working and if so is there a clearer explanation available with regards to the setup steps? I am not in a position to start with the server from scratch just to get discourse working and with no guarantees.
  • In Discourse behind reverse proxy and https - #2 by itsbhanusharma @Dark Matter provided the following as an apache2 solution, but I do not believe this solution has a node express environment in mind and with my localhost:8000 not resolving I am not sure if this would work:
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName  discourse.example.com
  DocumentRoot /website/discourse

  RewriteEngine On
  ProxyPreserveHost On
  ProxyRequests Off
  ProxyPass / unix:/var/discourse/shared/socket-only/nginx.http.sock|http://localhost/
  ProxyPassReverse  / unix:/var/discourse/shared/socket-only/nginx.http.sock|http://localhost/
  ErrorLog /var/log/apache2/discourse.error.log
  LogLevel warn
  CustomLog /var/log/apache2/discourse.access.log combined

  RewriteCond %{SERVER_NAME} =discourse.example.com
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
  • Is the issue here with my discourse config? Is the issue perhaps my apache routing (I suspect)? Or am I missing something more fundamental here?
  • I do think that resolving this will help a lot of users in the future.
  • First prize would be to route to discourse on the server locally.
  • Then routing via the url would be the next prize.

Best regards
Matthew Lucas

This is too complicated for me to understand completely, so consider this more of a wild guess than a solution. If the issue is that you’re not getting something to connect to your running docker container via localhost, you might try using the Docker IP. It might also be possible to use the web socket template and use the socket rather than the port. Some argue this is preferable for a number of reasons.

MKJ’s Opinionated Discourse Deployment Configuration might be useful to read.

@pfaffman@Matthew_Lucas is already using the socketed template, though…

I’m definitely using localhost for my external proxy, and that should work with the expose setting. I wouldn’t expect it to need to use the docker IP. And I wouldn’t expect this to fail only with Apache in front of the docker container for that reason.

You might need header configuration — see Add an offline page to display when Discourse is rebuilding or starting up - #2 by codinghorror

I have been doing my best to forget Apache configuration, but this section of the nginx configuration is something you would want to figure out how to replicate with Apache, I think.

    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Real-IP $remote_addr;

In particular, I don’t think it works right without the Host header.

Since you aren’t doing HTTPS, you should be able to use tcpdump to capture packet traces to see exactly what is going wrong.

However why put Apache in there at all? It’s just an extra hoop. If I were trying something like this, I’d be setting the Amazon load balancer to talk directly to port 8000 on the EC2 instance. I assume that you are terminating HTTPS on the ELB anyway. I suspect that their load balancer knows the normal headers to add without you needing to specify them, but if not obviously go ahead and set them. Just make sure that the port is accessible via the firewall.

I doubt I have much more of value to contribute here, but hopefully some of it helps you with your configuration. Best of luck.

1 Like

INSTALLATION OF DOCKER AND DISCOURSE ON AWS EC2 LINUX WITH EXISTING HTTPD REVERSE PROXY

Hi All

I have figured it out. I went back to the drawing board and started fresh with clean docker and discourse install. My process is detailed below:

Assumptions

• You already have an AWS EC2 Linux instance
• You can access your EC2 instance using SSH
• Apache 2 is already configured and running on your server

ACCESS YOUR EC2 INSTANCE USING SSH

  1. Open CMD / Terminal

  2. SSH into AWS EC2 instance

Example of screen once connected to EC2 instance

image

PERFORM SYSTEM UPDATES

  1. Perform system updates by running the following command in the console / terminal

$ sudo yum update

INSTALL DOCKER

  1. Search for the AWS Docker package by running the following command in the console / terminal

$ sudo yum search docker

  1. Get version information by running the following command in the console / terminal

$ sudo yum info docker

  1. Install docker by running the following command in the console / terminal

$ sudo yum install docker

  1. Add group membership for the default ec2-user so you can run all docker commands without using the sudo command by running the following command in the console / terminal
$ sudo usermod -a -G docker ec2-user
$ id ec2-user

# Reload a Linux user's group assignments to docker without logout

$ newgrp docker
  1. Install docker-compose by running the following command in the console / terminal
# 1. Get pip3 (python installer package)
$ sudo yum install python3-pip
 
# 2. Then run any one of the following
$ sudo pip3 install docker-compose # with root access
 
# OR
 
$ pip3 install --user docker-compose # without root access for security reasons

# Check permissions
$ sudo chmod -v +x /usr/local/bin/docker-compose
  1. Enable docker service to start automatically on boot by running the following command in the console / terminal

$ sudo systemctl enable docker.service

  1. Start the docker service by running the following command in the console / terminal

$ sudo systemctl start docker.service

  1. Check the docker service is running by running the following command in the console / terminal, the output will look similar to this

Basic docker commands that are good to know :

# Docker version
$ docker version

# Docker compose version
$ docker-compose version

# Start Docker service
$ sudo systemctl start docker.service

# Stop Docker service
$ sudo systemctl stop docker.service

# Restart Docker service
$ sudo systemctl restart docker.service

# Docker service status
$ sudo systemctl status docker.service

INSTALL DISCOURSE

  1. Create discourse folder

$ sudo mkdir /var/discourse

  1. Clone the discourse docker image to the newly created directory

$ sudo git clone https://github.com/discourse/discourse_docker.git /var/discourse

  1. Copy the standalone.yml file to the containers folder

$ cp /var/discourse/samples/standalone.yml /var/discourse/containers/discourse.yml

  1. Edit the new discourse.yml file as follows, (the important bits exposed ports for http and email details for smtp, modify in the file don’t copy and paste)

Editing options:

1. You can use nano and edit via command line / terminal
2. On AWS if you are using MATE with PLUMA, you can use PLUMA to edit the file
3. You can use WINSCP to SSH into the machine and edit the file via the Windows GUI

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  ## Uncomment the next line to enable the IPv6 listener
  #- "templates/web.ipv6.template.yml"
  - "templates/web.ratelimited.template.yml"
  ## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "8080:80"   # http (run on local port 8080, docker port 80)
  #- "443:443" # https (make sure this is commented out)

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  #db_shared_buffers: "256MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  #UNICORN_WORKERS: 3

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: 'faq.mobiloan.io'

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: ‘ enter email here ’

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: ENTER SMTP HERE
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: ENTER USERNAME HERE
  DISCOURSE_SMTP_PASSWORD: ENTER PASSWORD HERE
  DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true, we use Amazon SES)

 

  #DISCOURSE_SMTP_DOMAIN: discourse.example.com    # (required by some providers)
  #DISCOURSE_NOTIFICATION_EMAIL: noreply@discourse.example.com    # (address to send notifications from)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com
  
  ## The maxmind geolocation IP address key for IP address lookup
  ## see https://meta.discourse.org/t/-/137387/23 for details
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"
  1. Once the file is edited and saved, rebuild the discourse app using the following command in the terminal

$ /var/discourse/launcher rebuild discourse

Note if your yaml file is called something else then you would replace discourse with your yaml file name

Note that the rebuild takes time.

  1. One your app has been rebuilt the app should be running on port 8080. To verify this run the following command in the terminal

$ sudo lsof -i -P -n | grep LISTEN

Example output of ports

Notice docker is listening on port 8080, httpd (apache) is listening on port 80.

In terms of traffic routing apache will receive the web request and where appropriate will be routed to the discourse container on port 8080.

To make sure the routing happens you must configure the virtual host in the apache conf file.

APACHE VIRTUAL HOST CONFIGURATION

  1. The apache conf file is located at /etc/httpd/conf.d/00-virtualhosts.conf. httpd / apache / apache2 are different apache versions. Your files might be in different locations if you are not using the httpd service but the virtual host config in this case will be the same.

Editing options:

1. You can use nano and edit via command line / terminal
2. On AWS if you are using MATE with PLUMA, you can use PLUMA to edit the file
3. You can use WINSCP to SSH into the machine and edit the file via the Windows GUI

Edit the /etc/httpd/conf.d/00-virtualhosts.conf file by adding the following:

# FAQ (DISCOURSE ROUTES)

<VirtualHost *:80>
  	ServerName  sub.domain.com
  	ProxyPreserveHost On
    ProxyPass "/" "http://localhost:8080/"
    ProxyPassReverse "/" "http://localhost:8080/"
</VirtualHost>
  1. You need to create the relevant CNAME DNS record having the sub domain pointing to your AWS server / load balancer.

Our setup is a little complicated as we have our domain provider who points routes all traffic to an AWS load balancer.

The load balancer handles our SSL/TLS through an AWS provided certificate.

The loan balancer also routes traffic to the AWS instance.

Apache is installed on the AWS instance and routes the traffic to our:

  • Node express applications
  • Docker / discourse application.

CONCLUSION

This is only one solution, but it is the one that worked with our existing setup.

A big thank you to:

@Kane York for his article on Discourse,
@Axel Fernandes’ article on medium his notes on downloading the necessary discourse files and
@Vivek Gite from nixCraft and his article on cyberciti on installing Docker on AWS Linux 2.