Lots of warnings about rsyslogd, locks up discourse

Installation
1

I just installed Discourse on a Rackspace server, using the same general instructions as for Digital Ocean. No issues with the install, everything works great until I restart the server, at which point this starts appearing in my logs and eventually crashes the server completely, requiring a hard reboot to restore it.

rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.
Pidfile (and pid) already exist.

I have no idea where to start with this issue… help?

1 Like
2

Ah, I just noticed that the install instructions recommend Ubuntu 14, not 12. I will try upgrading the host OS to latest and report back.

3

No luck with Ubuntu 14, still getting the same errors after using ./launcher restart app:

2014-05-05 03:36:50 UTC LOG:  database system is ready to accept connections
supervisor pid: 39 unicorn pid: 47
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.
Pidfile (and pid) already exist.
config/unicorn_launcher: line 36: kill: (47) - No such process
config/unicorn_launcher: line 10: kill: (47) - No such process
exiting
ok: run: redis: (pid 14) 1s
ok: run: postgres: (pid 16) 1s
ok: run: copy_env: (pid 18) 1s
supervisor pid: 55 unicorn pid: 57
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.
Pidfile (and pid) already exist.

These last two lines repeat every second until I restart the host server. Weirdly, it only throws warnings when the app is restarted, not on the first run.

EDIT: actually that last statement seems to be wrong. After rebooting the host, start app also shows the same errors in logs. I’ll try to use the app and see if it hard locks the machine again.

4

What is the output of docker info and docker version?

5

as requested

% docker info
Containers: 2
Images: 11
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 15
Execution Driver: native-0.1
Kernel Version: 3.13.0-24-generic
WARNING: No swap limit support

and

% docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

the app has now been running for about 12 hours, and the logs are piling up, but still working.

% ./launcher logs app | grep rsyslogd | wc -l
WARNING: No swap limit support
42956
6

OK, I’m getting the same thing. Looks like the PID files aren’t getting cleaned up on container startup:

root@dhost:~# cat /var/run/rsyslogd.pid 
23
root@dhost:~# ps -p 23
  PID TTY          TIME CMD
   23 ?        00:00:00 nginx

We should clean up pid files, etc. on startup. Looking into it.

1 Like
7

What are we using rsyslogd for btw?

Also, I feel our logging story for nginx/pg/cron/ssh should be much improved, for one we are throwing away all logs on container rebuild. Perhaps we should shift /var/logs out of the container?

8

nginx logs, redis logs, general daemon logs.

Totally agree! Unfortunately, blocked by:

Here’s a preview of THE FUTURE:

It’ll let us safely clean up some stale state from the container before firing everything up.

1 Like
9

Was this resolved? Can this be closed?

10

Yeah, this should be fine