Docker install allows nginx log files to keep growing?

Bug
1

Is there a way to configure (e.g. via containers/app.yml) the maximum disk space the nginx log files are allowed to take up?

Here’s what I have

> sudo du -hs /shared/log/var-log/nginx/*
0       /shared/log/var-log/nginx/access.log
5.9G    /shared/log/var-log/nginx/access.log.1
5.7M    /shared/log/var-log/nginx/access.log.2.gz
8.0K    /shared/log/var-log/nginx/access.log.3.gz
13M     /shared/log/var-log/nginx/access.log.4.gz
1.2M    /shared/log/var-log/nginx/access.log.5.gz
5.8M    /shared/log/var-log/nginx/access.log.6.gz
80K     /shared/log/var-log/nginx/access.log.7.gz
0       /shared/log/var-log/nginx/error.log
44K     /shared/log/var-log/nginx/error.log.1
4.0K    /shared/log/var-log/nginx/error.log.2.gz
4.0K    /shared/log/var-log/nginx/error.log.3.gz
4.0K    /shared/log/var-log/nginx/error.log.4.gz
4.0K    /shared/log/var-log/nginx/error.log.5.gz
4.0K    /shared/log/var-log/nginx/error.log.6.gz
4.0K    /shared/log/var-log/nginx/error.log.7.gz

I installed via the standard instructions here. The site has been running with about 100 users for about 9 weeks.

4 Likes
2

I know @sam has changed the Docker container a fair bit around logging recently and there were bugs here that may have allowed logs to get too big…

Did you pull the latest container, e.g. SSH into your instance and

cd /var/discourse
git pull
./launcher rebuild app
4 Likes
3

Thanks. I’ve done that and I’ll keep my eye on the logs.

1 Like
4

I’ve found this as well, it seems nginx is failing to rotate the log. Googling has told me it’s likely due to permission issues on the log file itself but I haven’t been able to work out exactly what is happening.

5

grr we tried to fix this again and it appears that @supermathie’s fix did not take.

Will get it sorted.

4 Likes
6

Our forum just crashed due to no free disk space. It was caused by a 5GB /var/log/nginx/access.log.1 as described above.

Is this something that is fixed if I pull and rebuild the image?

Details:

I had an extra 5GB log file on the “physical” server too (from the world accessible nginx server which proxies to the Docker image) and I’ve solved the problem there by updating the logrotate config to force a rotation if the log file exceeds 100MB and running logrotate manually.

I’m not really sure how to solve this on the docker instance though. If I were to edit the logrotate config, would it be overwritten when I update Discourse?

I think I’d just like to add size 100M to the file /etc/logrotate.d/nginx. This would cause the file to be rotated every day or when it reaches 100M (whichever comes first), as long as logrotate is being called often enough to notice the file is >100M.

danny:/var/docker$ ./launcher ssh app
Warning: Permanently added '[0.0.0.0]:2222' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.2.0-54-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
root@discourse:~# cd /var/log/nginx
root@discourse:/var/log/nginx# ls -lahS
total 5.1G
-rw-r-----  1 www-data adm    5.1G Apr 10 14:04 access.log.1
-rw-r--r--  1 root     root    20M Mar  7 04:00 access.log.2.gz
-rw-r--r--  1 root     root    52K Mar  6 16:53 error.log.1
-rw-r-----  1 www-data adm     41K Apr 10 13:37 error.log
drwxr-x---  2 www-data adm    4.0K Mar  7 05:11 .
drwxrwxr-x 11 root     syslog 4.0K Apr 10 14:04 ..
-rw-r-----  1 www-data adm       0 Mar  7 05:11 access.log
root@discourse:/var/log/nginx# cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}
5 Likes
7

Yeah we got to fix this @supermathie any idea what is bust?

8

Given the file dates and size of the logs, the file was rotated but nginx didn’t reopen it.

As to how that happened… will need to investigate.

9

Fixed per:

https://github.com/discourse/discourse_docker/commit/5d256035c6c2c8685b8735141539c7e3bf835a74

to update

cd /var/discourse
git pull
./launcher rebuild app
4 Likes