How to troubleshoot sidekiq?


(Allen - Watchman Monitoring) #1

Along with my backup issues, sidekiq is failing on one of my instances.

Rebuilding hasn’t helped so far… what else can I look at?

Is this because there’s only 1GB on my DO Droplet? 1 GB Memory / 30 GB Disk / NYC2

Sidekiq heartbeat test failed, restarting

/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:74:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:35:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:445:in `warn'
config/unicorn.conf.rb:129:in `check_sidekiq_heartbeat'
config/unicorn.conf.rb:146:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:295:in `join'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

Backup issue - undefined method `every' for Jobs::CreateBackup:Class
(Allen - Watchman Monitoring) #2

There’s plenty of free space

# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       487M  8.0K  487M   1% /dev
tmpfs                      100M  384K  100M   1% /run
/dev/disk/by-label/DOROOT   30G  8.6G   20G  31% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                       498M  1.1M  496M   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        30G  8.6G   20G  31% /var/lib/docker/aufs/mnt/660c80352081dd3afbb6ac720f772a77b49a2f4e674c204a29c427d8bfabbf02
shm                         64M     0   64M   0% /var/lib/docker/containers/660c80352081dd3afbb6ac720f772a77b49a2f4e674c204a29c427d8bfabbf02/shm

and a 2gb swap file is in place, I can only assume it’s being used right.

ls -al /swapfile

-rw------- 1 root root 2147483648 May 24 2014 swapfile


(Sam Saffron) #3

Not sure you have swap setup right, what does free return


(Allen - Watchman Monitoring) #4

Here we go

# free
             total       used       free     shared    buffers     cached
Mem:       1017956     904552     113404      27372       8296      78204
-/+ buffers/cache:     818052     199904
Swap:      2097148      30696    2066452

(Sam Saffron) #5

What happens if you change this to a 2GB instance? Do the issues go away?

What version of Docker are you running? What version of Discourse? What does docker info and docker version return.


(Allen - Watchman Monitoring) #6

I ran poweroff on the droplet, upgraded and now docker won’t start.

» 20 Dec 2015 02:32:43.683 Dec 20 03:32:43 www kernel: [51211.619198] type=1400 audit(1450600363.517:661): apparmor=“DENIED” operation=“ptrace” profile=“docker-default” pid=19299 comm=“ps” requested_mask=“trace” denied_mask=“trace” peer=“docker-default” www/syslog

This is the latest test-passed discourse, Docker version 1.9.1, build a34a1d5 and docker info fails at the moment.

root@www:/var/discourse# docker -D -d
Warning: '-d' is deprecated, it will be removed soon. See usage.
WARN[0000] please use 'docker daemon' instead.          
DEBU[0000] docker group found. gid: 999                 
DEBU[0000] Server created for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported 
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported 

(Allen - Watchman Monitoring) #7

the kernel in my droplet was out of sync with the os

I trued that up with a power off, changing the kernel in DO and powering on.

now I can get docker info

/var/discourse/shared# docker info
Containers: 1
Images: 0
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-74-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 1.955 GiB
Name: www.my-forum-hostname.com
ID: VNCR:DBKB:ZDEK:DBNB:3E6X:Y5MY:V6Y3:FIIJ:GZ7X:XPU2:CHAC:H43H
WARNING: No swap limit support

Trying to rebuild or bootstrap my discourse results in

# ./launcher bootstrap app


WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Error response from daemon: could not find image: no such id: cb7b58c22b114085db98b08bf974d8dd809e62955567e527c18e56857384c7a7
Your Docker installation is not working correctly

See: Docker error on Bootstrap

is that that topic relevant to docker 1.9.1?


(Allen - Watchman Monitoring) #8

tl;dr - yes, but I also ended up completely resetting docker along the way.

I’m just going to leave this at 2gb, as I intend to have healthy amounts of traffic here soon.


(Allen - Watchman Monitoring) #9

While testing incoming email, I see the sidekiq error is back

Sidekiq heartbeat test failed, restarting

/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:74:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/logger.rb:35:in `add'
/usr/local/lib/ruby/2.0.0/logger.rb:445:in `warn'
config/unicorn.conf.rb:129:in `check_sidekiq_heartbeat'
config/unicorn.conf.rb:146:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:295:in `join'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'