502 bad gateway error when running alongside nginx


(Bobby) #1

I am getting a 502 bad gateway error while running discourse alongside linux. It is currently the only website I have running on my machine.

Here is the error I get when I use tail -f /var/log/nginx/error.log:

2015/04/08 21:03:13 [error] 8436#0: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 173.245.54.87, server: epicdinosaurz.net, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "epicdinosaurz.net"

Here is my disco.conf in nginx:

types {
    text/csv csv;
}

upstream discourse { server 127.0.0.1:3000; }

proxy_cache_path /var/nginx/cache keys_zone=one:10m max_size=200m;
map $http_x_forwarded_proto $thescheme {
  default $scheme;
  https https;
}

limit_req_zone $binary_remote_addr zone=flood:10m rate=13r/s;
limit_req_zone $binary_remote_addr zone=bot:10m rate=222r/m;
limit_req_status 429;
server {

  listen 80;
  gzip on;
  gzip_vary on;
  gzip_min_length 987;
  gzip_comp_level 5;
  gzip_types application/json text/css application/x-javascript application/javascript;

  server_name epicdinosaurz.net;
  server_tokens off;
  sendfile on;
  keepalive_timeout 66;
  client_max_body_size 10m ;
  set $public /var/www/discourse/public;
  add_header X-UA-Compatible "IE=edge";
  etag off;

  location ^~ /backups/ {
    internal;
  }

  location / {
    root $public;
    add_header ETag "";

  location ~* \.(eot|ttf|woff|ico)$ {
      expires 1y;
      add_header Cache-Control public;
      add_header Access-Control-Allow-Origin *;
     }

    location ~ ^/assets/ {
      expires 1y;
      gzip_static on;
      add_header Cache-Control public;
      break;
    }

    location ~ ^/plugins/ {
      expires 1y;
      add_header Cache-Control public;
    }

    location ~ /_?emoji/ {
      expires 1y;
      add_header Cache-Control public;
    }

    location ~ ^/uploads/ {

      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $thescheme;
      proxy_set_header X-Sendfile-Type X-Accel-Redirect;
      proxy_set_header X-Accel-Mapping $public/=/downloads/;
      expires 1y;
      add_header Cache-Control public;

      location ~ /stylesheet-cache/ {
          try_files $uri =404;
      }
      location ~* \.(gif|png|jpg|jpeg|bmp|tif|tiff)$ {
          try_files $uri =404;
      }
      location ~ /_optimized/ {
          try_files $uri =404;
      }

      proxy_pass http://discourse;
      break;
    }

    location ~ ^/admin/backups/ {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $thescheme;
      proxy_set_header X-Sendfile-Type X-Accel-Redirect;
      proxy_set_header X-Accel-Mapping $public/=/downloads/;
      proxy_pass http://discourse;
     break;
    }

    try_files $uri @discourse;
  }

  location /downloads/ {
    internal;
    alias $public/;
  }

  location @discourse {
  limit_req zone=flood burst=13 nodelay;
  limit_req zone=bot burst=103 nodelay;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $thescheme;
    proxy_pass http://discourse;
  }

}

and here is my discourse.conf in /var/www/discourse:

# Discourse supports multiple mechanisms for production config, detailed in the _defaults file.
# This file includes the most commonly changed enviroment variables, to allow users following the official install guide to quickly get started; if the option you're looking for is not here it may be defined in the _defaults file.

# Change the following!
# All settings apply to production only

# database name running discourse, in INSTALL-ubuntu this is discourse_prod
db_name = discourse_prod

# hostname running the forum; i.e the external address of your form.
hostname = "www.epicdinosaurz.net"

# address of smtp server used to send emails. If testing, this can be left empty; but e-mails are a very important part of discourse.
smtp_address = smtp.mandrillapp.com 

# port of smtp server used to send emails. This varies based on your e-mail host.
smtp_port = 587

# domain passed to smtp server
smtp_domain =

# username for smtp server
smtp_user_name = notify@epicdinosaurz.net

# password for smtp server
smtp_password = Not gonna put this here for the world to see ;)

# enable TLS encryption for smtp connections
smtp_enable_start_tls = true

# enable MiniProfiler for developers, by default this is true. This shows response times in the top left for admin users only. Uncomment this line to turn this off.
# load_mini_profiler = false

# recommended, cdn used to access assets. This can be left empty at first.
cdn_url =

# comma delimited list of emails that have developer level access. 
developer_emails = email redacted

# More advanded settings. Unless you changed these in your enviroment, the defaults will work.

## Database

# host address for db server, uncomment if needed
# db_host = localhost

# port running db server, uncomment if needed
# db_port = 5432

# username accessing database, if connecting remotely
# db_username = discourse

# password used to access the db, if connecting remotely
# db_password =

# Redis

# redis server address
# redis_host = localhost

# redis server port
# redis_port = 6379

# redis password
# redis_password =

(Jens Maier) #2

Officially, the only installation method supported here is this: discourse/INSTALL-cloud.md at master · discourse/discourse · GitHub

Inofficially:

I’m assuming you’re running Discourse through rails server with Thin or WebRICK when instead you should be using Unicorn:

LD_PRELOAD=/usr/lib/libjemalloc.so.1 \
RUBY_GC_MALLOC_LIMIT=90000000 \
RAILS_ENV=production \
UNICORN_SIDEKIQS=1 \
UNICORN_WORKERS=3 \
bundle exec unicorn -D -c config/unicorn.conf.rb

If you’re running under Ruby 2.2, also add UNICORN_ENABLE_OOBGC=0 to the environment. Edit config/unicorn.conf.rb if you need to fine-tune which local IP unicorn binds to.


(Bobby) #3

@elberet I am using unicorn; but more particularly this script here: Sign in to GitHub · GitHub

I’ve had no issues with it and I think it is some sort of misconfiguration on my part.

I added UNICORN_ENABLE_OOBGC=0 to config/unicorn.conf.rb, now how do I set which IP unicorn binds to?


(Jens Maier) #4

First of all, you should only add the OOBGC hack if you are using Ruby 2.2 or later. On all earlier Ruby versions, omitting that variable will reduce performance. Also, if you’re editing unicorn.config.rb, change the 1 in ENV['UNICORN_ENABLE_OOBGC'] ||= "1" to a zero instead. (Again: Ruby 2.2+ only!)
You can configure the bind address by editing the listen line, e.g. listen "127.0.0.1:3000".

Second, as I said, this method is unsupported for a reason. There are quite a few moving parts between incoming HTTP connections and Discourse, and that script is tweaking all the knobs

Here are a few things you may want to check:

  • Firewall: iptables -S – is the INPUT chain rejecting nginx’s packages?
  • Unicorn: netstat -lnpt |grep unicorn – is it really listening on the port it should? Is it accidentally listening on an IPv6 address (i.e. the first column reads tcp6)?
  • Error logs: less /var/www/discourse/log/unicorn.stderr.log – also see unicorn.stdout.log. Are there any errors during unicorn’s startup?

(Bobby) #5

Here are the only interesting startup logs. They’re from about an hour ago when installing an addon.

I, [2015-04-08T22:10:44.568442 #23440]  INFO -- : Refreshing Gem list
/var/www/discourse/plugins/plugins/blog/plugin.rb:11: warning: already initialized constant BLOG_HOST
/var/www/discourse/plugins/blog/plugin.rb:11: warning: previous definition of BLOG_HOST was here
/var/www/discourse/plugins/plugins/blog/plugin.rb:12: warning: already initialized constant BLOG_DISCOURSE
/var/www/discourse/plugins/blog/plugin.rb:12: warning: previous definition of BLOG_DISCOURSE was here
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:437:in `add_route': Invalid route name, already in use: 'blog'  (ArgumentError)
You may have defined two routes with the same name using the `:as` option, or you may be overriding a route already defined by a resource with the same naming. For the latter, you can restrict the routes created with `resources` as explained here: 
http://guides.rubyonrails.org/routing.html#restricting-the-routes-created
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:1495:in `add_route'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:1472:in `decomposed_match'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:1453:in `block in match'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:1443:in `each'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:1443:in `match'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/mapper.rb:536:in `mount'
	from /var/www/discourse/plugins/plugins/blog/plugin.rb:67:in `block (2 levels) in activate!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:344:in `instance_exec'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:344:in `eval_block'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:359:in `block in clear!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:359:in `each'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.1.9/lib/action_dispatch/routing/route_set.rb:359:in `clear!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:35:in `block in clear!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:33:in `each'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:33:in `clear!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:15:in `reload!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:26:in `block in updater'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.9/lib/active_support/file_update_checker.rb:75:in `call'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.1.9/lib/active_support/file_update_checker.rb:75:in `execute'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:27:in `updater'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/routes_reloader.rb:7:in `execute_if_updated'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application/finisher.rb:71:in `block in <module:Finisher>'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/initializable.rb:30:in `instance_exec'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/initializable.rb:30:in `run'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/initializable.rb:55:in `block in run_initializers'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:150:in `block in tsort_each'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:183:in `block (2 levels) in each_strongly_connected_component'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:219:in `each_strongly_connected_component_from'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:182:in `block in each_strongly_connected_component'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:180:in `each'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:180:in `each_strongly_connected_component'
	from /home/discourse/.rvm/rubies/ruby-2.0.0-p643/lib/ruby/2.0.0/tsort.rb:148:in `tsort_each'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/initializable.rb:54:in `run_initializers'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/application.rb:300:in `initialize!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/railtie.rb:194:in `public_send'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.1.9/lib/rails/railtie.rb:194:in `method_missing'
	from /var/www/discourse/config/environment.rb:5:in `<top (required)>'
	from config.ru:3:in `require'
	from config.ru:3:in `block in <main>'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:55:in `instance_eval'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:55:in `initialize'
	from config.ru:1:in `new'
	from config.ru:1:in `<main>'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn.rb:48:in `eval'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn.rb:48:in `block in builder'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:764:in `call'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:764:in `build_app!'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:137:in `start'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

(Sam Saffron) #6

not needed anymore, commented that out yesterday

in other news meta has been running on 2.2 for the last 15 hours or so.


(Bobby) #7

I know that docker is the official install method, but do you have any thoughts about how to fix this bad gateway problem? I’ve had no problems without docker until trying to install a plugin and restarting :sweat:


(Sam Saffron) #8

We do not officially support any install except for the Docker method.


(Kane York) #9

For this reason. Too many variables without it…


(system) #10