Nginx Bad Gateway - hitmanforum


(ampburner) #1

Hi,

Yesterday my Digital Ocean instance stopped working. I had not made any changes to configuration, nor had I updated recently. It just stopped working.

I logged on to the server to find that:

=> / is using 95.2% of 39.25GB
=> There is 1 zombie process

After reading this thread 502 Bad Gateway - nginx issue

I ran

./launcher cleanup
Starting cleanup (bytes free 0)
Finished Cleanup (bytes free 827856)

./launcher start app

Then the instance came online again.

Is this simply an symptom of my droplet running out of disk space or are there other things that I should investigate?

What are my options - it seems to me that 40Gb ought to be enough to run a discourse install (roughly 2 years and counting).

Are there ways to archive old discussions, file uploads, or other temp files.

Please advise.

update:
followed @codinghorror’s tips here
https://meta.discourse.org/t/discourse-using-almost-95-of-40gb-total-drive-space-is-this-a-problem/43966/2?u=ampburner&source_topic_id=48586

still lots of usage.

Ncdu:

root@discourse:~# cd /var/discourse
root@discourse:/var/discourse# ncdu
ncdu 1.10 ~ Use the arrow keys to navigate, press ? for help
--- /var/discourse/shared/standalone --------------------------------------------------------
                        /..
    7.1GiB [##########] /backups
    6.7GiB [######### ] /uploads
    3.7GiB [#####     ] /postgres_data
    1.5GiB [##        ] /log
  798.6MiB [#         ] /postgres_backup
  376.0MiB [          ] /vendor_bundle
  295.7MiB [          ] /postgres_dataX
  217.3MiB [          ] /redis_data
   37.0MiB [          ] /postgres_data_new
  112.0KiB [          ] /postgres_run
   28.0KiB [          ] /state
 Total disk usage:  20.7GiB  Apparent size:  20.5GiB  Items: 96163

(ampburner) #2

My var/lib/docker/aufs folder is 22 GB. What is all that stuff?

--- /var ------------------------------------------------------------------------------------
                        /..
   22.2GiB [##########] /lib
   13.6GiB [######    ] /discourse
  206.5MiB [          ] /cache
  170.0MiB [          ] /log
    1.2MiB [          ] /backups
   32.0KiB [          ] /spool
e   4.0KiB [          ] /tmp
e   4.0KiB [          ] /opt
e   4.0KiB [          ] /mail
e   4.0KiB [          ] /local
e   4.0KiB [          ] /crash
@   0.0  B [          ]  lock
@   0.0  B [          ]  run

 Total disk usage:  36.2GiB  Apparent size:  36.2GiB  Items: 295529

(Jeff Atwood) #3
  • You can remove /postgres_backup as that is a backup from the postgres 9.3 to 9.5 upgrade

  • Not sure what /postgres_dataX is? I’d remove that too.

  • You can clear /log, probably.

  • You’re holding 7GB of /backups which on a 40GB drive is a lot. Move those to S3 or somewhere else.

You may need to enter the Docker container and do ncdu there. Remember you’re looking at the host operating system here, which is usually enough, but maybe there’s something haywire on the pseudo-VM in your Docker container with disk space.


(ampburner) #4

:S

this happened again. I removed everything you suggested and that helped for a while.

is it a problem if my var/lib/docker folder is very large?

Are there any structural changes that I could make to prevent this?


(Epic47Sides) #5

HI @ampburner !!

I’m getting a 502 Bad Gateway error while connecting to hitmanforum.

I guess this is server problems. Can you send me a topic link concerning this problem please?

Thanks for help :slight_smile:


(ampburner) #6

@Epic47Sides I think the problem is still disk space.

I have cleaned up logs as suggested, but the diskspace fills up again. I suspect there is something wrong inside the docker container but I am unsure how to investigate.


(ampburner) #7


(Sam Saffron) #8

Yeah junk all the nginx logs and be sure to rebuild

We had an issue with previous image where it would not cycle logs


(Matt Palmer) #9

That looks like (a) your nginx logs aren’t stored in the data volume like they’re supposed to, and (b) they’re not getting rotated (might be related to (a)). What have you done to your container configuration to break it like that?


(ampburner) #10

@mpalmer I have no idea,I think I have set the server, container etc up 2 years ago following a guide posted on the Discourse blog.

I’m not aware of making any specific config changes to place the log file outside the data volume, but it’s possilble that I just have a very old install.

What would I need to do to put it in the correct place?


(Epic47Sides) #11

I can’t help you since i have no knowledge about hosting a server.

However i’m wishing you good luck hoping to see the forum back soon :slight_smile:

Cheers!!


(Matt Palmer) #12

If you don’t have the skills available to bring the setup up-to-date and up-to-spec, I’d be inclined to recommend taking a backup of the site, building a whole new container (as described in this post and its descendants for some discussion on that), and loading the restore into the new container.


(Jeff Atwood) #13

If you are willing to purchase the $99 one-time setup from https://discourse.org/buy we could fix it up for you if that helps.


(ampburner) #14

ah, looks related to

my file looks like

root@discourse-app:/etc/logrotate.d# less nginx
/var/log/nginx/*.log {
  daily
  missingok
  rotate 14
  compress
  delaycompress
  notifempty
  create 0640 www-data www-data
  sharedscripts
  postrotate
    sv 1 nginx
  endscript
}

which seems up to date.
so I think @mpalmer is right, the logs are saved to the wrong folder.

/var/discourse/templates

 - file:
     path: /etc/runit/1.d/00-fix-log-permissions
     chmod: "+x"
     contents: |
       #!/bin/bash
       mkdir -p /var/log/nginx
       chown -R www-data:www-data /var/log/nginx
       chown www-data:www-data /var/log/nginx
       chown -f syslog:adm /var/log/syslog*
       chown -f syslog:adm /var/log/auth.log*
       chown -f syslog:adm /var/log/kern.log*

(ampburner) #15

hah, this happened again (during my holiday to Spain).

The obvious reason is that I failed to take long-term action and had kept manually deleting log files and backup files (but not regularly enough)

If no one this thread has a DIY silver bullet fix, then I will consider the buy discourse for $100 option. I’ll have to look into it first :slight_smile:

thanks!