SSL + AWS ELB >> 503 Service Unavailable (Back-end server is at capacity)

docker

(Andrew Stroup) #1

Back again :slight_smile: and as always, apologies for my noob devops questions, but here it goes …

I’m trying to setup https for a docker instance of Discourse that’s deployed on AWS. I’m able to successfully run Discourse on AWS using the following setup:

  • EC2 t2.medium instance (attached to an elastic IP address)
  • RDS postgres db.t2.small (SSD / 100GB)
  • ElastiCache Redis server cache.m3.medium (2 clusters)
  • SES
  • appropriate VPC and Security Groups setup for SSH, HTTP, HTTPS, Postgres (5432), and Redis (6379)

I recently tried to setup HTTPS for the Discourse server and have ran into a few issues, hoping someone has either seen before or I can get help resolving:

I have my key and certs and following AWS’ tutorial, I created an Elastic Load Balancer (to include HTTPS setup), where I uploaded the certs. Some things to note:

  • I had to setup the Health Check to TCP:80/ because the root URL redirects to /login if not logged in (this is just a note for others trying to do this).

Discourse docker app doesn’t support listening (at least for ELB) on HTTPS (443).

  • I setup two Listeners: (Load Balancer Protocol > Instance Protocol)
  • HTTP (80) > HTTP (80)
  • HTTPS (443) > HTTPS (443) >> which includes referencing the SSL cert and cipher

NOTE: This does NOT work. I have to change the second Listener to HTTPS (443) > HTTP (80) even though I’ve gone into the EC2 server (cd /var/docker/containers sudo nano app.yml) and added in the “443:443” to the expose: section.

Rebuilding the app (sudo ./launcher rebuild app) breaks ELB, causing:

  • the Health Check to fail and returns a 503 Service Unavailable (Back-end server is at capacity)
  • to resolve, requires the EC2 instance to be removed from the Load Balancer, EC2 instance reboot and reattach the instance to the ELB

I found a thread regarding the 503 error, not sure if it’s 100% the same, but may be helpful for troubleshooting, but this is out of my realm of expertise. >> AWS Developer Forums: ELB 503 Service Unavailable: Back-end ...

I started to follow @sam’s SSL + Docker setup (Allowing SSL / HTTPS for your Discourse Docker setup), but I may be missing something because I can’t find the folder

/var/discourse/shared/standalone/ssl/

instead when I cd / ls through the directory, I get

/var/discourse/shared/web-only/

with no path to an ssl folder within web-only (the folders available are: backups, log, state, uploads), which is when I stopped and decided to start this thread.

With all this said …

If I have everything configured as noted above in AWS, the outstanding issues that still remain are:

  • the Discourse server doesn’t let the ELB listen on HTTPS (443) even if you set “443:443” in app.yml
  • I don’t have a good resolution on how to perform updates that don’t break the ELB, forcing me to detach, reboot, and reattach the EC2 instance
  • I can’t get HTTP to redirect to HTTPS (am I missing something here?)

I know there’s the setting in Security (below) that sets Discourse.base_url, but that doesn’t seem to change any actions regarding a redirect from HTTP to HTTPS

Any help on this would be awesome. I feel like I’ve gotten 90% of the way there, just need help identifying how to resolve these last couple of issues :smile:


(Michael Scott Shappe) #2

If you’re using HTTPS via ELB, you do not need to use it within the Docker or Discourse. Have your ELB back-end to HTTP/80 and let it handle all the HTTPS proxying itself.

Put another way: as far as Discourse and Docker are concerned, they’re still just HTTP. ELB will act as a secure proxy, using the key you set up under AWS.


(Andrew Stroup) #3

Thanks @mshappe that answers one of the problems!

The remaining two are:

  • what’s the best method to redirect HTTP to HTTPS using AWS ELB?
  • any thoughts on how to address updates / rebuild of the Discourse server breaking the ELB?

Thanks!


(Ker Ruben Ramos) #4

How about not thinking it as “redirect http->https using ELB”?

Redirect it on the end point. Apache’s htaccess, Nginx https redirect, or App’s response header redirect.

I hope that helps.


(Michael Scott Shappe) #5

For the redirect, I think that https setting should work, but I haven’t tried it. I know for a straight rails application, configuring nginx or Rails itself to always redirect to https just works, provided that the URL being redirected to is the ELB’s URL, not your server’s. In general, that should be true for your setup–the URL of your service is now the ELB, not your EC2 instance.

As for upgrades “breaking” ELB, I think you’re just not being patient. ELB requires, by default, 10 “good” hits on its test URL before it will resume thinking of an instances as “good”.


(Kane York) #6

Make the directory. Although, actually, as of today, that directory will be made for you now.

Also, I believe that various research indicated that spying is being done on unencrypted backend connections. You should have SSL all the way through, even if on self-signed certs as long as the cert sha-2 fingerprint is verified.


(Hugh Evans) #7

I like to have the whole domain fronted by a CDN such a CloudFront or CloudFlare and then you can use that to redirect HTTP > HTTPS.


(Dean Taylor) #8

There is actually a /srv/status end point which should return a simple 200 “ok” response.


(Andrew Stroup) #9

So I ended up just ditching the ELB as there’s other questions about how to setup duplicate EC2 instances to actually use load balancing and manage asset paths and updates appropriately. I was able to setup HTTPS within Discourse/docker and used an elastic IP.

So I was able to get this setup, BUT I used

/var/discourse/shared/web-only/ssl/

INSTEAD of

/var/discourse/shared/standalone/ssl/ as I wasn't sure standalone would work.

Thanks for the clarification!

Fair enough, I had some difficulty setting this up and ended up ditching ELB (see above) and just used CloudFront as an asset CDN on a different domain. Basically, I thought I had it setup correctly, but when I’d ping the correct URL which was pointed to an A record ALIAS of the CloudFront, it’d redirect to the ELB URL.

Thanks! I’ll give this a try instead of the TCP:80/ health check.


(Kevin Wildenradt) #10

Here is a tutorial (apparently written by a 14 year old) on setting up Discourse with AWS, including ELB.

https://web.archive.org/web/20140424075917/http://0ak.org/how-to-setup-discourse-on-amazon-aws/


#11

Thanks for this link. I can’t see where his post talks about ELB or am I missing something?