Discourse has stopped working after LetsEncrypt is enabled


(Rohit Manglik) #1

Hi,

I am following the guidelines at Setting up Let's Encrypt.

After I rebuilt the app, site stopped opening. Can you please guide of how to fix the error.

Rebuild Log: Dropbox - Link not found

app.yml:

ubuntu@ip-172-31-0-252:/var/discourse/containers$ cat app.yml
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  - "templates/web.ssl.template.yml"
  - "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "80:80"   # http
  - "443:443" # https

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "256MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 4

  ## TODO: The domain name this Discourse instance will respond to
  DISCOURSE_HOSTNAME: iitjee.edugorilla.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'hello@edugorilla.com'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  DISCOURSE_SMTP_ADDRESS: smtp.sparkpostmail.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: SMTP_Injection
  DISCOURSE_SMTP_PASSWORD: .........................................
  DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  LETSENCRYPT_ACCOUNT_EMAIL: hello@edugorilla.com

  ## The CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: //discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
	  host: /var/discourse/shared/standalone
	  guest: /shared
  - volume:
	  host: /var/discourse/shared/standalone/log/var-log
	  guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
	- exec:
		cd: $home/plugins
		cmd:
		  - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  - exec: rails r "SiteSetting.notification_email='hello@edugorilla.com'"
  - exec: echo "End of custom commands"

Logs:

run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/00-fix-var-logs
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
run-parts: executing /etc/runit/1.d/enable-brotli
run-parts: executing /etc/runit/1.d/letsencrypt
[Tue Jun 20 17:58:39 UTC 2017] Domains not changed.
[Tue Jun 20 17:58:39 UTC 2017] Skip, Next renewal time is: Sat Aug 19 16:38:53 UTC 2017
[Tue Jun 20 17:58:39 UTC 2017] Add '--force' to force to renew.
[Tue Jun 20 17:58:39 UTC 2017] Installing key to:/shared/ssl/iitjee.edugorilla.com.key
[Tue Jun 20 17:58:39 UTC 2017] Installing full chain to:/shared/ssl/iitjee.edugorilla.com.cer
[Tue Jun 20 17:58:39 UTC 2017] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Tue Jun 20 17:58:39 UTC 2017] Reload error for :
Started runsvdir, PID is 215
sh: echo: I/O error
ok: run: redis: (pid 223) 0s
ok: run: postgres: (pid 222) 0s
				_._
		   _.-``__ ''-._
	  _.-``    `.  `_.  ''-._           Redis 3.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 223
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
	  `-._    `-.__.-'    _.-'
		  `-._        _.-'
			  `-.__.-'

223:M 20 Jun 17:58:39.333 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
223:M 20 Jun 17:58:39.333 # Server started, Redis version 3.0.6
223:M 20 Jun 17:58:39.333 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
223:M 20 Jun 17:58:39.333 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
223:M 20 Jun 17:58:39.351 * DB loaded from disk: 0.018 seconds
223:M 20 Jun 17:58:39.352 * The server is now ready to accept connections on port 6379
supervisor pid: 224 unicorn pid: 238
2017-06-20 17:58:39.503 UTC [241] LOG:  database system was shut down at 2017-06-20 17:58:08 UTC
2017-06-20 17:58:39.508 UTC [241] LOG:  MultiXact member wraparound protections are now enabled
2017-06-20 17:58:39.523 UTC [222] LOG:  database system is ready to accept connections
2017-06-20 17:58:39.524 UTC [245] LOG:  autovacuum launcher started

(Jay Pfaffman) #2

Your DNS records resolve to two different IP addresses:

$ dig +short iitjee.edugorilla.com
104.27.148.171
104.27.149.171

Do both of them point to your server? If not, Let’s Encrypt is probably hitting the wrong one and won’t give you the certs you need.


(Rafael dos Santos Silva) #3

This is CloudFlare proxy, disable it if you want to use Let’s Encrypt (the orange cloud on the DNS entry MUST be gray).


(Rohit Manglik) #5

Thanks @Falco and @pfaffman for quick response.

I changed the DNS entry to Gray and rebuild the app but still I am getting same error.

site845774@server:~$ dig +short iitjee.edugorilla.com
35.154.203.196

New Console Log: Dropbox - Link not found


(Jay Pfaffman) #6

Well, I see this:

HTTP 429 Too Many Requests https://index.rubygems.org/info/http_accept_language
HTTP GET https://index.rubygems.org/info/jmespath
Bundler::HTTPError: Net::HTTPTooManyRequests: <html>
<head><title>429 Too Many Requests</title></head>
<body bgcolor="white">
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx</center>
</body>
</html>

But I’m wondering if you ahve some firewall on port 80 and 443?


(Rohit Manglik) #7

It’s a Vanilla install in AWS Instance. Is there any settings I can check on AWS? I made DNS pointer to grey in CloudFlare, I did not explicitly create any firewall.


(Jay Pfaffman) #8

It would seem like if CloudFlare could get to your site, then you could. But it also seems like if anyone could access your site without going through cloudflare, what would be the point?

You might try telnet localhost 80 from your host and see what you get (you might type ‘get /’ if it appears that you’ve connected).

Then you can look in to AWS security groups and see if you’re letting port 80 through. Or maybe you’re letting 80 through, but not 443

That’s probably it:

wget -Y off -O /dev/null -S -v http://iitjee.edugorilla.com
--2017-06-20 13:34:22--  http://iitjee.edugorilla.com/
Resolving iitjee.edugorilla.com (iitjee.edugorilla.com)... 35.154.203.196
Connecting to iitjee.edugorilla.com (iitjee.edugorilla.com)|35.154.203.196|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 301 Moved Permanently
  Server: nginx/1.11.6
  Date: Tue, 20 Jun 2017 20:34:23 GMT
  Content-Type: text/html
  Content-Length: 185
  Connection: keep-alive
  Location: https://iitjee.edugorilla.com/
Location: https://iitjee.edugorilla.com/ [following]
--2017-06-20 13:34:23--  https://iitjee.edugorilla.com/
Connecting to iitjee.edugorilla.com (iitjee.edugorilla.com)|35.154.203.196|:443... ^C

It’s redirecting to 443, but 443 isn’t answering.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html


(Rohit Manglik) #9

I received following reply

ubuntu@ip-172-31-0-252:/var/discourse$ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get /
HTTP/1.1 400 Bad Request
Server: nginx/1.11.6
Date: Tue, 20 Jun 2017 20:34:48 GMT
Content-Type: text/html
Content-Length: 173
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.11.6</center>
</body>
</html>
Connection closed by foreign host.

(Jay Pfaffman) #10

You can try telnet localhost 443. If it hangs, it’s definitely a firewall thing. If it’s connection refused then it’s something wrong locally.


(Rohit Manglik) #11

Yes, you are right. I fixed it just now. Rebuilding. - I will be back with result.


(Jay Pfaffman) #12

If I was right, you didn’t need to rebuild.


(Rohit Manglik) #13

It’s working now! Thanks a lot @pfaffman.

Going back to email issue: Upgrade, force SSL, Emails not working on Bitnami install :slight_smile: