Discourse Crashing


(Michael Henderson) #1

hi, and i appreciate and welcome any feedback! Our discourse instance crashes every once in a while and we can not figure out what is causing it. Our discourse is updated to the latest version and just for the record it was created with the one click install from digital ocean running on 4gigs of RAM. please help. here is a shot of the logs . This is urgent so please ask questions if you need to and i will provide as much info as i can! Thanks so much!


(Jeff Atwood) #2

Make sure you are on the very latest version.


#3

Does your droplet have a swapfile?


(Sam Saffron) #4

can you explain what you mean by “crashing”? What happens? What does memory on the box look like in that state? What plugins do you have running?


(Quincy Larson) #5

@codinghorror yes we are upgraded to the latest release: v1.6.0.beta11 +58

@sam At first we experience 500 errors when trying to reply to topics / create topics. After a while, forum.freecodecamp.com just returns a blank screen.

We experienced this about a month ago, and we “solved” it by moving from a 2GB RAM Digital Ocean box to a 4GB one.

Another detail that might be helpful: we originally installed Discourse using Digital Ocean’s one-click install.

Also, we have not made any customizations to Discourse at all outside of tweaking options available in the admin panel.

@ron_jeremy where would a swapfile normally be located? We can chec


(Sam Saffron) #6

Upgrade again to +80, we had some issues late last week with the tests passed branch

If it “white screen” or errors with 500 I need to see logs and memory status in the droplet


#7

In the Linux terminal, run:

free -h

Your swapfile will be listed if you have one. You can create one if you don’t see it listed (you will need admin privs):

In the below example, we will be creating a 4GB swapfile file:

sudo fallocate -l 4G /swapfile

Now we enable the file:

sudo chmod 600 /swapfile

sudo mkswap /swapfile

sudo swapon /swapfile

Now we make the swapfile permanent:

sudo nano /etc/fstab

At the bottom of the file add the following line entry:

/swapfile   none    swap    sw    0   0

If you run free -h again you should see the swapfile


(Michael Henderson) #8

@sam @ron_jeremy @codinghorror wow, thanks so much for the quick response, it is greatly appreciated!

@sam we updated a couple of days ago and it shows v1.6.0.beta11 +58. When we do an update does it not pull you to the latest version? Just want to get as much info as I can for future events.

Also, when it “crashed” yesterday, we knew this by going to our URL and being presented with a white screen. Power cycling the server seem to have fixed it. Would you be okay with me sending you logs to your inbox? Our logs could contain sensitive information.

@ron_jeremy we will check up on the swap to see if it is enabled.

All, Like I said in the beginning, thanks for the great response! You guys are awesome and we truely appreciate it! Happy coding :smile:


(Michael Henderson) #9

Here is a screenshot of the most recent logs, I hope this helps. If you notice anything out of the ordinary please let me know. Also, we can provide more information if need be. Thanks again for all your help !

Notice the fatal alerts in this screenshot:

we also have a lot of spoof attack alerts that are critical, i did not upload a screenshot because it contains ip addresses.


(Quincy Larson) #10

When you say “upgrade again to 80+” the admin panel is showing I am already up to date, and when I go into the repo and pull master, there has only been one commit since July 7

commit cad2919f12b414b229b8c3e2a56baa5c3f4e40a3
Author: Guo Xiang Tan <tgx_world@hotmail.com>
Date:   Fri Jul 15 15:22:15 2016 +0800

    Add config options to skip assignment of a mac address.

How do I go about manually upgrading to 80+?


(Quincy Larson) #11

Discourse just crashed again.

I restarted and created a swap file as you instructed. Now when I run free -h I see:

root@discourse-chani:/var/discourse# free -h
             total       used       free     shared    buffers     cached
Mem:          3.9G       3.4G       512M       275M        31M       1.5G
-/+ buffers/cache:       1.8G       2.0G
Swap:         2.0G         0B       2.0G

Does this look like it’s properly configured? So once the memory fills up it will start using swap, and this will prevent further crashes?


(Jeff Atwood) #12

Go to /admin/upgrade and upgrade from there to make sure you are on latest.


(Sam Saffron) #13

Be sure to upgrade this thing from the console at least once

cd /var/discourse
git pull
./launcher rebuild app

(Quincy Larson) #14

OK - we already were on latest (I upgraded a few days ago):


(Sam Saffron) #15

Did you upgrade via the web UI, or from the command line?

If it gets into a crashed state in the next 8 hours can you PM me I can log in and have a look.


(Quincy Larson) #16

I usually upgrade via the web UI, but sometimes I will do it from the command line by doing a git pull and restarting. It looks like we are currently only one commit behind master, and next time I restart, that commit will be live as well.

By the way, here are our memory use stats a few hours after a restart. Does this look accurate? Should the system be reclaiming RAM?


(Michael Henderson) #17

Our forum completely crashed again yesterday. We are trying to refrain from touching the files within the container but could this have something to do with unicorn workers ? Do you have any other suggestions as to what we can check to provide you with information to make an educated guess on what is causing the issue? I have searched a lot and can’t seem to dig anything up. Thanks for being patient with me! :slight_smile:


(Quincy Larson) #18

We’re continuing to crash about once every 24 hours.

Here are Discourse’s logs from our crash an hour ago:

I’m PM’ing you as well about this.


(Jeff Atwood) #19

Nobody ever answered this? @ossia @michaelhenderson we URGENTLY need to know the answer to this.

If you are using plugins that discourse does not ship as default please turn them off and rebuild immediately. That is step ZERO.


(Michael Henderson) #20

basically, the only thing we have done to discourse is, install it on a droplet using Digital Ocean’s “One-Click Application Install”. As for plugins, i know that i have not installed any and i don’t think @ossia has installed any but i will let him speak for himself.

Here is a screen shot of plugins that are installed :slight_smile: