Discourse won't load after rebuild


(James Taylor) #1

Hi, guys. Not sure if this is the right place to post this issue but it seems relevant and related…

I’ve set up a forum website for a client, it’s been live since January. Just renewed the SSL certificates about an hour ago. But now that I’ve restarted my Digital Ocean machine, rebuilt the app, and restarted it… it’s giving me an error like you talk about here: cannot load app.

I removed an AdSense plugin, based on the previous discussion. Rebuilt the app, still gives the cannot load app error when visiting the site.

Anyone have any ideas on what’s causing this? Willing to provide auxiliary info as needed. Thanks!


"Cannot load app" for a meta topic
(Rafael dos Santos Silva) #2

Can you share the URL?

The rebuild doesn’t error at all? Can you share the logs?


(James Taylor) #3

Absolutely. The URL is:

forums.dissidianerviciatournaments.com

The rebuild doesn’t have any errors. Not that I saw as it was rebuilding, anyway. I will share the logs. For that part though, where would I go to retrieve them? I haven’t had any problems (and thus no need to check them) until now.


(Rafael dos Santos Silva) #4

Just rebuild again to get new logs.


(James Taylor) #5

It’s rebuilding again now. I’ve told Linux to redirect output to a text file. Will post momentarily. Thank you.


(James Taylor) #6

Ok. So, after dealing with some non related technical issues around my home, I was able to get back on here and putty into the web server, told it to rebuild and appended &> to the end to obtain a text file. Used WinSCP to pull the file, and here it is. I went through it with Notepad using Ctrl+F to look for “err” and the only things I could find seemed trivial, non related, and like they had absolutely no impact on the build itself. Here’s the snippet:

2018-04-12 17:59:53.403 UTC [44] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2018-04-12 17:59:53.403 UTC [44] LOG:  listening on IPv6 address "::", port 5432
2018-04-12 17:59:53.405 UTC [44] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2018-04-12 17:59:53.419 UTC [47] LOG:  database system was shut down at 2018-04-12 17:59:43 UTC
2018-04-12 17:59:53.423 UTC [44] LOG:  database system is ready to accept connections
I, [2018-04-12T17:59:58.384624 #13]  INFO -- : 
I, [2018-04-12T17:59:58.384903 #13]  INFO -- : > su postgres -c 'createdb discourse' || true
2018-04-12 17:59:58.459 UTC [57] postgres@postgres ERROR:  database "discourse" already exists
2018-04-12 17:59:58.459 UTC [57] postgres@postgres STATEMENT:  CREATE DATABASE discourse;
createdb: database creation failed: ERROR:  database "discourse" already exists
I, [2018-04-12T17:59:58.461202 #13]  INFO -- : 
I, [2018-04-12T17:59:58.461626 #13]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2018-04-12 17:59:58.542 UTC [68] postgres@discourse ERROR:  role "discourse" already exists
2018-04-12 17:59:58.542 UTC [68] postgres@discourse STATEMENT:  create user discourse;
ERROR:  role "discourse" already exists
I, [2018-04-12T17:59:58.544671 #13]  INFO -- : 
I, [2018-04-12T17:59:58.545098 #13]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
I, [2018-04-12T17:59:58.626593 #13]  INFO -- : GRANT
...
...
...
I, [2018-04-12T17:59:59.108176 #13]  INFO -- : File > /var/lib/postgresql/take-database-backup  chmod: +x
I, [2018-04-12T17:59:59.111142 #13]  INFO -- : File > /var/spool/cron/crontabs/postgres  chmod: 
I, [2018-04-12T17:59:59.111477 #13]  INFO -- : > echo postgres installed!
I, [2018-04-12T17:59:59.113472 #13]  INFO -- : postgres installed!

So we can see that postgres was successful and all privileges were modified as needed. The only “error” was that upon checking for an install of discourse, the installer found discourse already available, and so it didn’t re-install. That’s my understanding.

The only other mentions of the string “err” in the log are from records of output to other files, and the error template for when there are too many HTTP requests.

Do you think I should swim through the warnings for clues? I can’t upload the text file here because, new users can’t upload attachments. If you’d like I can Google Drive or Dropbox it.


(Bhanu Sharma) #7

Can You post the last 10 lines of the log to start with?


(James Taylor) #9

I’ll post the last 20ish just in case.

[Thu Apr 12 18:05:30 UTC 2018] Installing cron job
[Thu Apr 12 18:05:30 UTC 2018] Good, bash is found, so change the shebang to use bash as preferred.
[Thu Apr 12 18:05:30 UTC 2018] OK

I, [2018-04-12T18:05:30.398889 #13]  INFO -- : > cd /root/acme.sh && LE_WORKING_DIR="${LETSENCRYPT_DIR}" ./acme.sh --upgrade --auto-upgrade
I, [2018-04-12T18:05:31.152523 #13]  INFO -- : [Thu Apr 12 18:05:30 UTC 2018] Installing from online archive.
[Thu Apr 12 18:05:30 UTC 2018] Downloading https github(dot)com/Neilpang/acme.sh/archive/master.tar.gz
[Thu Apr 12 18:05:30 UTC 2018] Extracting master.tar.gz
[Thu Apr 12 18:05:30 UTC 2018] Installing to /shared/letsencrypt
[Thu Apr 12 18:05:30 UTC 2018] Installed to /shared/letsencrypt/acme.sh
[Thu Apr 12 18:05:30 UTC 2018] Installing alias to '/root/.profile'
[Thu Apr 12 18:05:30 UTC 2018] OK, Close and reopen your terminal to start using acme.sh
[Thu Apr 12 18:05:30 UTC 2018] Good, bash is found, so change the shebang to use bash as preferred.
[Thu Apr 12 18:05:31 UTC 2018] OK
[Thu Apr 12 18:05:31 UTC 2018] Install success!
[Thu Apr 12 18:05:31 UTC 2018] Upgrade success!

I, [2018-04-12T18:05:31.155697 #13]  INFO -- : File > /etc/nginx/letsencrypt.conf  chmod: 
I, [2018-04-12T18:05:31.160913 #13]  INFO -- : File > /etc/runit/1.d/letsencrypt  chmod: +x
I, [2018-04-12T18:05:31.161449 #13]  INFO -- : Replacing (?-mix:ssl_certificate.+) with ssl_certificate /shared/ssl/$$ENV_DISCOURSE_HOSTNAME.cer;
 in /etc/nginx/conf.d/discourse.conf
I, [2018-04-12T18:05:31.162090 #13]  INFO -- : Replacing (?-mix:#?ACCOUNT_EMAIL=.+) with ACCOUNT_EMAIL=$$ENV_LETSENCRYPT_ACCOUNT_EMAIL
 in /shared/letsencrypt/account.conf
I, [2018-04-12T18:05:31.162615 #13]  INFO -- : Replacing (?-mix:ssl_certificate_key.+) with ssl_certificate_key /shared/ssl/$$ENV_DISCOURSE_HOSTNAME.key;
 in /etc/nginx/conf.d/discourse.conf
I, [2018-04-12T18:05:31.163215 #13]  INFO -- : Replacing (?-mix:add_header.+) with add_header Strict-Transport-Security 'max-age=63072000'; in /etc/nginx/conf.d/discourse.conf
I, [2018-04-12T18:05:31.163755 #13]  INFO -- : Replacing (?m-ix:add_header Referrer-Policy 'no-referrer-when-downgrade';) with add_header Referrer-Policy 'no-referrer-when-downgrade';
add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain in /etc/nginx/conf.d/discourse.conf
I, [2018-04-12T18:05:31.164335 #13]  INFO -- : > echo "Beginning of custom commands"
I, [2018-04-12T18:05:31.166752 #13]  INFO -- : Beginning of custom commands

I, [2018-04-12T18:05:31.167378 #13]  INFO -- : > echo "End of custom commands"
I, [2018-04-12T18:05:31.169449 #13]  INFO -- : End of custom commands

I, [2018-04-12T18:05:31.170395 #13]  INFO -- : Terminating async processes
I, [2018-04-12T18:05:31.170889 #13]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main pid: 44
I, [2018-04-12T18:05:31.171522 #13]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 160
160:signal-handler (1523556331) Received SIGTERM scheduling shutdown...
2018-04-12 18:05:31.171 UTC [44] LOG:  received fast shutdown request
2018-04-12 18:05:31.175 UTC [44] LOG:  aborting any active transactions
2018-04-12 18:05:31.180 UTC [44] LOG:  worker process: logical replication launcher (PID 53) exited with exit code 1
2018-04-12 18:05:31.183 UTC [48] LOG:  shutting down
2018-04-12 18:05:31.214 UTC [44] LOG:  database system is shut down
160:M 12 Apr 18:05:31.239 # User requested shutdown...
160:M 12 Apr 18:05:31.239 * Saving the final RDB snapshot before exiting.
160:M 12 Apr 18:05:31.270 * DB saved on disk
160:M 12 Apr 18:05:31.271 # Redis is now ready to exit, bye bye...
sha256:abee8f8e8c5c2598dea4bdd721a68218ca2c4f1e03f327c380a8b55032d4c24a
b1f65bc713cf5840a9f65176c5f24891502e86934d9fbb552cece1559be3a84d
Removing old container
+ /usr/bin/docker rm app
app

+ /usr/bin/docker run --shm-size=512m -d --restart=always -e LANG=en_US.UTF-8 -e RAILS_ENV=production -e UNICORN_WORKERS=2 -e UNICORN_SIDEKIQS=1 -e RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 -e RUBY_GC_HEAP_GROWTH_MAX_SLOTS=40000 -e RUBY_GC_HEAP_INIT_SLOTS=400000 -e RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.5 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e LETSENCRYPT_DIR=/shared/letsencrypt -e DISCOURSE_HOSTNAME=forums.dissidianerviciatournaments(dot)com -e DISCOURSE_DEVELOPER_EMAILS=dissidianerviciatournaments@mail(dot)com,darthmodulus@gmail(dot)com -e DISCOURSE_SMTP_ADDRESS=smtp.mailgun(dot)org -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=postmaster@forums.dissidianerviciatournaments(dot)com -e DISCOURSE_SMTP_PASSWORD=(hidden for security) -e LETSENCRYPT_ACCOUNT_EMAIL=dissidianerviciatournaments(at)mail(dot)com -h DNTForums-app -e DOCKER_HOST_IP=172.17.0.1 --name app -t -p 2045:80 -p 443:443 -v /var/discourse/shared/standalone:/shared -v /var/discourse/shared/standalone/log/var-log:/var/log --mac-address 02:fa:65:1f:e5:26 local_discourse/app /sbin/boot
5eccf1d64663f64f6ad603ce384d73d4fb64f5a6e06e5cf95e335833473b2390

(Rafael dos Santos Silva) #10

You have a custom install with a reverse proxy. In this case it’s not really related to Discourse :confused:

Also, if you are using a reverse proxy, why are using our let’s encrypt template? You can’t use both.


(Bhanu Sharma) #11

Nothing loads for http.
For https,I get the Good 'ol certificate expired warning.


(James Taylor) #12

Well, to be completely honest, I don’t know where it got that it was supposed to be a reverse proxy. When I finished setup back in January, I had used Let’s Encrypt and the related certbot commands to set up SSL for the machine. Both the www and non-www versions are now (re)certified. Clearly it’s only supposed to be using 443:443 for SSL… what would you suggest I do to remedy it? I’m assuming dig through the app.yaml and make an adjustment somewhere?


(Bhanu Sharma) #13

I’d suggest you configure your server to run behind an nginx proxy in sockets mode instead of using non-conventional ports.


(James Taylor) #14

Really, it’s frustrating to me because it worked just fine until today yesterday when the certificates expired. Unprofessional as it was, I accidentally let it lapse. So I went and fixed it today, and as soon as I stopped the container, re-certified, and rebuilt the container, the response as shown in the OP’s image is what is gotten now.


(Rafael dos Santos Silva) #15

Are you running another site in this server?

If you aren’t just do:

  1. stop / kill / remove you reverse proxy (nginx/apache/etc)

  2. Edit app.yml to use 80:80 again.

  3. Rebuild


(James Taylor) #16

I’ll give that a shot. Though, won’t editing app.yml to use 80:80 kill my SSL?

Edit: no, to answer your question, I’m not running another site on the server, just the forum. It is linked to a “main” site that is hosted elsewhere, but they’re completely separated.


(Rafael dos Santos Silva) #17

Nope, you need to listen in port 80 and 443 for a functional website.


(Joe Buhlig) #18

For a completely different solution:

When things like this crop up, I spin up a new server and restore a backup. It eliminates all the issues I had with the previous installation. The DNS propagation delay is minor compared to the time it takes for me to work through all the possible solutions.


(James Taylor) #19

Since I’m already neck deep in attempting to fix, if all else fails, this is what I’ll try. Thanks!


(James Taylor) #20

So, I’ve edited the app.yml, it doesn’t anywhere state that I should have anything but 80:80 and 443:443 open. Those are the only ones exposed. I used the ps command and looked through my list of running processes, I don’t see anything else running… but for good measure I uninstalled nginx anyways, by using the purge command and then autoremove. Restarted my droplet. Double checked the running processes again, rebuilt the app, and it’s still giving me the same thing… --name app -t -p 2045:80.

What the heck am I missing?? :stuck_out_tongue:


(Rafael dos Santos Silva) #21

Can you paste your app.yml here (redact passwords & emails)?