Does the Docker installation still rotate logs?

Support
1

Hi,

We’ve been using Discourse on one box for a while and things have been running smoothly. I regularly rebuild when we change Git plugins and once a month to ensure a good container and fewer upgrade problems.

I was checking logs today and saw:

root@disc:/var/discourse# ls -lh shared/standalone/log/var-log/nginx/
total 98G
-rw-r--r-- 1 www-data www-data  742 Oct 22  2016 access.letsencrypt.log
-rw-r----- 1 www-data www-data  97G May 17 02:03 access.log
-rw-r----- 1 www-data www-data 318M Aug 10  2016 access.log.1
-rw-r----- 1 www-data www-data  61M Aug  1  2016 access.log.10.gz
-rw-r----- 1 www-data www-data  57M Jul 31  2016 access.log.11.gz
-rw-r----- 1 www-data www-data  53M Jul 30  2016 access.log.12.gz
-rw-r----- 1 www-data www-data  52M Jul 29  2016 access.log.13.gz
-rw-r----- 1 www-data www-data  55M Jul 28  2016 access.log.14.gz
-rw-r----- 1 www-data www-data  50M Aug 11  2016 access.log.1.gz
-rw-r----- 1 www-data www-data  55M Aug  9  2016 access.log.2.gz
-rw-r----- 1 www-data www-data  59M Aug  8  2016 access.log.3.gz
-rw-r----- 1 www-data www-data  59M Aug  7  2016 access.log.4.gz
-rw-r----- 1 www-data www-data  54M Aug  6  2016 access.log.5.gz
-rw-r----- 1 www-data www-data  57M Aug  5  2016 access.log.6.gz
-rw-r----- 1 www-data www-data  57M Aug  4  2016 access.log.7.gz
-rw-r----- 1 www-data www-data  58M Aug  3  2016 access.log.8.gz
-rw-r----- 1 www-data www-data  60M Aug  2  2016 access.log.9.gz
-rw-r--r-- 1 www-data www-data    0 Sep 30  2016 error.letsencrypt.log
-rw-r----- 1 www-data www-data  22M May 16 13:00 error.log
-rw-r----- 1 www-data www-data 1.2K Aug 10  2016 error.log.15.gz

As this was a box just for Discourse, I didn’t have as close monitoring as I normally would have, but do now.

I’ve deleted those logs now, but can anyone confirm if log rotation is still working for the Docker installation, and explain why it stopped working?

Cheers

Sam

1 Like
2

What’s the version of the docker image your have? (output of cd /var/discourse && git rev-parse HEAD)

2 Likes
3

Hi Rafael,

Thanks for the prompt response.

My installation is based on:

commit 88ee2e35b59b7e9c6f7390c138b0fc7526a823cb
Author: Guo .......... (no-spam)
Date:   Mon Apr 17 11:11:56 2017 +0800

    Add support for Ruby 2.4 in 1.4.0.
4

I noticed this on a site I was debugging today. I did a rebuild a couple days ago, I’ll check if the logs rotated and get the version information tomorrow.

2 Likes
5

My, what a large log you have…

1 Like
6

shared/standalone/log/var-log/

No such folder exists on one older installation I checked. Should I be concerned about that @sam?

From a more modern install I did in March:

jeff@forum:/var/discourse$ ls -lh shared/standalone/log/var-log/nginx/
total 2.0G
-rw-r----- 1 www-data www-data    0 Mar 13 01:03 access.letsencrypt.log
-rw-r----- 1 www-data www-data  865 Mar 12 14:17 access.letsencrypt.log.1
-rw-r--r-- 1 www-data www-data  363 Mar  6 13:06 access.letsencrypt.log.2.gz
-rw-r----- 1 www-data www-data 520M May 16 23:06 access.log
-rw-r----- 1 www-data www-data 544M May 16 00:51 access.log.1
-rw-r----- 1 www-data www-data  69M May  7 00:57 access.log.10.gz
-rw-r----- 1 www-data www-data  74M May  6 00:46 access.log.11.gz
-rw-r----- 1 www-data www-data  72M May  5 00:38 access.log.12.gz
-rw-r----- 1 www-data www-data  67M May  4 00:55 access.log.13.gz
-rw-r----- 1 www-data www-data  68M May  3 00:42 access.log.14.gz
-rw-r----- 1 www-data www-data  63M May 15 00:50 access.log.2.gz
-rw-r----- 1 www-data www-data  66M May 14 00:46 access.log.3.gz
-rw-r----- 1 www-data www-data  74M May 13 00:39 access.log.4.gz
-rw-r----- 1 www-data www-data  77M May 12 00:55 access.log.5.gz
-rw-r----- 1 www-data www-data  77M May 11 00:53 access.log.6.gz
-rw-r----- 1 www-data www-data  74M May 10 00:52 access.log.7.gz
-rw-r----- 1 www-data www-data  74M May  9 00:55 access.log.8.gz
-rw-r----- 1 www-data www-data  66M May  8 00:35 access.log.9.gz
-rw-r----- 1 www-data www-data    0 Mar 13 01:03 error.letsencrypt.log
-rw-r----- 1 www-data www-data  453 Mar 12 14:17 error.letsencrypt.log.1
-rw-r--r-- 1 www-data www-data  249 Mar  6 13:06 error.letsencrypt.log.2.gz
-rw-r----- 1 www-data www-data 270K May 16 20:34 error.log
-rw-r----- 1 www-data www-data 627K May 15 20:59 error.log.1
-rw-r----- 1 www-data www-data  20K May  7 00:21 error.log.10.gz
-rw-r----- 1 www-data www-data  17K May  6 00:04 error.log.11.gz
-rw-r----- 1 www-data www-data  12K May  4 21:47 error.log.12.gz
-rw-r----- 1 www-data www-data  14K May  3 23:22 error.log.13.gz
-rw-r----- 1 www-data www-data 9.5K May  3 00:15 error.log.14.gz
-rw-r----- 1 www-data www-data 3.4K May 15 00:41 error.log.2.gz
-rw-r----- 1 www-data www-data 8.5K May 13 23:58 error.log.3.gz
-rw-r----- 1 www-data www-data  15K May 12 21:50 error.log.4.gz
-rw-r----- 1 www-data www-data 4.4K May 12 00:44 error.log.5.gz
-rw-r----- 1 www-data www-data  12K May 11 00:51 error.log.6.gz
-rw-r----- 1 www-data www-data  22K May 10 00:52 error.log.7.gz
-rw-r----- 1 www-data www-data 3.9K May  8 22:38 error.log.8.gz
-rw-r----- 1 www-data www-data 7.8K May  7 23:44 error.log.9.gz

The access.log appears to only rotate at 500mb?

1 Like
7

I just checked a recent install and can confirm we are indeed rotating.

The log rotations depends on a few things:

First and foremost, being on a recent image.

This is how you can find what you are on:

cd /var/discourse
git pull
nano launcher

Look for this line in launcher https://github.com/discourse/discourse_docker/blob/master/launcher#L67

eg:

image=discourse/discourse:1.3.10

Then compare to the image you are running:

./launcher enter app
cat /VERSION
1.3.10

If you are an old image, do this ASAP:

cd /var/discourse
./launcher rebuild app

Make sure you have /var/log mounted to host

Our sample template contains the lines:

volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

This says that /var/log in the container will be mounted at /var/discourse/shared/standalone/log/var-log

If this is missing, add it asap we expect the logs to live outside of the container, not inside it.

How our log rotation is configured?

Our web template contains this info

  - file:
     path: /etc/logrotate.d/nginx
     contents: |
        /var/log/nginx/*.log {
          daily
          missingok
          rotate 14
          compress
          delaycompress
          notifempty
          create 0640 www-data www-data
          sharedscripts
          postrotate
            sv 1 nginx
          endscript

Once a day we rotate logs for NGINX and compress old logs.

The image runs cron (actually anacron on boot and then cron which is a bit better)

./launcher enter app
ps aux | grep cron

root        48  0.0  0.0   4244   764 ?        Ss   May04   0:00 runsv cron
root        51  0.0  0.0  29300  2956 ?        S    May04   0:01 cron -f

cron runs logrotate daily in /etc/cron.daily

cat /var/log/syslog.1
May 17 07:30:01 server anacron[13014]: Will run job `cron.daily' in 5 min.
May 17 07:35:01 server  anacron[13014]: Job `cron.daily' started
May 17 07:35:01 server  anacron[13359]: Updated timestamp for job `cron.daily' to 2017-05-17
May 17 07:52:41 server  anacron[13014]: Job `cron.daily' terminated

So yeah, this is all working but you may need a rebuild and you may need to add the missing share.

5 Likes
8

It looks to me like this site is properly configured, but logs are still not getting removed.

var-log has syslog.6.gz dated November 2(Install was Oct 30).

I rebuilt 4 days ago.

root@:/var/discourse/shared/standalone# du -hs /var/discourse/shared/standalone/log/*
234M    /var/discourse/shared/standalone/log/rails
7.3G    /var/discourse/shared/standalone/log/var-log
root@:/var/discourse# grep discourse/discourse launcher 
image=discourse/discourse:1.3.10
root@:/var/discourse# ./launcher enter app
root@-app:/var/www/discourse# cat /VERSION 
1.3.10


## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

Any ideas?

9

What does ls return for /var/discourse/shared/standalone/log/var-log/nginx

2 Likes
12

Although I have cleaned up now, ls -l showed last modification of the rotated logs to be May 2016.

I have rebuild the container a few times since then; so maybe something around that time introduced a regression.

Cheers

Sam

13

Oops. I wondered about that, sorry @sam . And now the power is out. I’m pretty sure those access and error logs rotate weekly.

Edit: but I was wrong:

# ls -l /var/discourse/shared/standalone/log/var-log/nginx
total 3379516
-rw-r----- 1 www-data www-data          0 May 15 08:05 access.letsencrypt.log
-rw-r--r-- 1 root     root          46994 May 13 17:07 access.letsencrypt.log.1
-rw-r----- 1 www-data www-data  158165198 May 18 17:08 access.log
-rw-r----- 1 www-data www-data 1249534339 May 18 08:05 access.log.1
-rw-r----- 1 www-data www-data   66684311 May  9 08:04 access.log.10.gz
-rw-r----- 1 www-data www-data   63257643 May  8 08:01 access.log.11.gz
-rw-r----- 1 www-data www-data   58825762 May  7 07:38 access.log.12.gz
-rw-r----- 1 www-data www-data   64048026 May  6 07:46 access.log.13.gz
-rw-r----- 1 www-data www-data   70102481 May  5 07:36 access.log.14.gz
-rw-r----- 1 www-data www-data  113561684 May 17 07:40 access.log.2.gz
-rw-r----- 1 www-data www-data  123669410 May 16 07:54 access.log.3.gz
-rw-r----- 1 www-data www-data  122622868 May 15 08:05 access.log.4.gz
-rw-r----- 1 www-data www-data  117469078 May 14 08:02 access.log.5.gz
-rw-r----- 1 www-data www-data   98707874 May 13 07:49 access.log.6.gz
-rw-r----- 1 www-data www-data   69591997 May 12 07:37 access.log.7.gz
-rw-r----- 1 www-data www-data   64107252 May 11 07:41 access.log.8.gz
-rw-r----- 1 www-data www-data   69511375 May 10 08:03 access.log.9.gz
-rw-r----- 1 www-data www-data          0 May 15 08:05 error.letsencrypt.log
-rw-r--r-- 1 root     root          55603 May 13 17:07 error.letsencrypt.log.1
-rw-r----- 1 www-data www-data       7337 May 18 10:03 error.log
-rw-r----- 1 www-data www-data  799197396 May 18 03:51 error.log.1
-rw-r----- 1 www-data www-data       2970 May  8 22:32 error.log.10.gz
-rw-r----- 1 www-data www-data      10125 May  8 05:19 error.log.11.gz
-rw-r----- 1 www-data www-data       2638 May  7 01:52 error.log.12.gz
-rw-r----- 1 www-data www-data       2605 May  6 01:38 error.log.13.gz
-rw-r----- 1 www-data www-data       1621 May  4 19:08 error.log.14.gz
-rw-r----- 1 www-data www-data   39658321 May 17 07:40 error.log.2.gz
-rw-r----- 1 www-data www-data   39805951 May 16 07:54 error.log.3.gz
-rw-r----- 1 www-data www-data   42393565 May 15 08:05 error.log.4.gz
-rw-r----- 1 www-data www-data   24644324 May 14 08:02 error.log.5.gz
-rw-r----- 1 www-data www-data    4818480 May 13 06:22 error.log.6.gz
-rw-r----- 1 www-data www-data       5947 May 12 07:23 error.log.7.gz
-rw-r----- 1 www-data www-data       3393 May 11 04:21 error.log.8.gz
-rw-r----- 1 www-data www-data       2685 May  9 15:32 error.log.9.gz
14

I just double checked and rotation looks good to me on a live site.

root@forum:/var/discourse/shared/standalone/log/var-log/nginx# ls -l --block-size=M
total 1199M
-rw-r----- 1 www-data www-data   0M Mar 13 01:03 access.letsencrypt.log
-rw-r----- 1 www-data www-data   1M Mar 12 14:17 access.letsencrypt.log.1
-rw-r--r-- 1 www-data www-data   1M Mar  6 13:06 access.letsencrypt.log.2.gz
-rw-r----- 1 www-data www-data  44M Aug 26 04:53 access.log
-rw-r----- 1 www-data www-data 426M Aug 26 01:03 access.log.1
-rw-r----- 1 www-data www-data  61M Aug 17 00:41 access.log.10.gz
-rw-r----- 1 www-data www-data  62M Aug 16 00:53 access.log.11.gz
-rw-r----- 1 www-data www-data  58M Aug 15 00:37 access.log.12.gz
-rw-r----- 1 www-data www-data  51M Aug 14 01:04 access.log.13.gz
-rw-r----- 1 www-data www-data  50M Aug 13 00:39 access.log.14.gz
-rw-r----- 1 www-data www-data  57M Aug 25 01:00 access.log.2.gz
-rw-r----- 1 www-data www-data  58M Aug 24 00:47 access.log.3.gz
-rw-r----- 1 www-data www-data  58M Aug 23 00:48 access.log.4.gz
-rw-r----- 1 www-data www-data  60M Aug 22 00:56 access.log.5.gz
-rw-r----- 1 www-data www-data  52M Aug 21 01:03 access.log.6.gz
-rw-r----- 1 www-data www-data  49M Aug 20 00:59 access.log.7.gz
-rw-r----- 1 www-data www-data  60M Aug 19 00:59 access.log.8.gz
-rw-r----- 1 www-data www-data  59M Aug 18 00:42 access.log.9.gz
-rw-r----- 1 www-data www-data   0M Mar 13 01:03 error.letsencrypt.log
-rw-r----- 1 www-data www-data   1M Mar 12 14:17 error.letsencrypt.log.1
-rw-r--r-- 1 www-data www-data   1M Mar  6 13:06 error.letsencrypt.log.2.gz
-rw-r----- 1 www-data www-data   0M Aug 26 01:03 error.log
-rw-r----- 1 www-data www-data   1M Aug 25 15:33 error.log.1
-rw-r----- 1 www-data www-data   1M Aug 16 15:10 error.log.10.gz
-rw-r----- 1 www-data www-data   1M Aug 15 22:11 error.log.11.gz
-rw-r----- 1 www-data www-data   1M Aug 14 22:18 error.log.12.gz
-rw-r----- 1 www-data www-data   1M Aug 13 20:20 error.log.13.gz
-rw-r----- 1 www-data www-data   1M Aug 12 11:40 error.log.14.gz
-rw-r----- 1 www-data www-data   1M Aug 24 23:13 error.log.2.gz
-rw-r----- 1 www-data www-data   1M Aug 23 13:34 error.log.3.gz
-rw-r----- 1 www-data www-data   1M Aug 22 17:38 error.log.4.gz
-rw-r----- 1 www-data www-data   1M Aug 21 17:57 error.log.5.gz
-rw-r----- 1 www-data www-data   1M Aug 20 20:44 error.log.6.gz
-rw-r----- 1 www-data www-data   1M Aug 20 00:48 error.log.7.gz
-rw-r----- 1 www-data www-data   1M Aug 18 10:57 error.log.8.gz
-rw-r----- 1 www-data www-data   1M Aug 17 14:02 error.log.9.gz

Note that there are a limited number of error.log and access.log files, going back to Aug 17th, and the daily log file is compressed into the .gz version, at 1/10th the size.

1 Like
15

Rotation is working again on new containers (built in August).

2 Likes