Tons of Redis Errors


(James Cook) #1

I’ve recently been seeing this in my log files.

Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
4 ActionView::Template::Error (MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk.
Commands that may modify the data set are disabled. Please check Redis logs for details about the error.)
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:199:in `call_pipelined' web-exception

Has anybody come across this recently?


Could sidekiq queue be reason for 500 errors?
Tags become unusable if used once and then topic is deleted
(James Cook) #2

This just happened again:

Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 
2 Failed to process job: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 

Stacktrace:

"/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:199:in `call_pipelined'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:157:in `block in call_pipeline'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:293:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:155:in `call_pipeline'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2299:in `block in multi'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `block in synchronize'", "/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2291:in `multi'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/backends/redis.rb:91:in `publish'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus.rb:233:in `publish'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus.rb:468:in `block in new_subscriber_thread'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/timer_thread.rb:98:in `do_work'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/timer_thread.rb:29:in `block in initialize'"]

Apologies, can’t format that correctly. This seems to happen when I’m using the console.


(Rafael dos Santos Silva) #3

Disk Space is plenty?


(James Cook) #4

Should be. 2GB Digital Ocean droplet created 21 days ago. Forum not being used yet.


(Rafael dos Santos Silva) #5

Can you run a df -h ?


(James Cook) #6
Filesystem                 Size  Used Avail Use% Mounted on
udev                       991M  4.0K  991M   1% /dev
tmpfs                      201M  348K  200M   1% /run
/dev/disk/by-label/DOROOT   40G   11G   27G  29% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                      1001M  588K 1001M   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        40G   11G   27G  29% /var/lib/docker/aufs/mnt/e01e39c23091db7cede7da8673b5a4efc30eaa5252247ecbe472e40aa28ccbdd
shm                         64M  4.0K   64M   1% /var/lib/docker/containers/5453b058278d67388a26b65f26daf0737b4c05e20a0add38a83d9282cb493246/shm

Thanks


(Matt Palmer) #7

You probably want to start here. No point going off on wild goose chases if the logs will tell you exactly what went wrong.


(James Cook) #8

Excuse my ignorance. Where can I find these logs? I can’t see them in the log folder. Thanks


(James Cook) #9

It doesn’t look like a logfile is set for Redis. Is it being caught somewhere else? I can’t see much when running ./launcher logs app


(Matt Palmer) #10

Redis should be logging to the container’s stdout, so the log entries should be in the output of ./launcher logs app. If they’re not, you’ve got a whole other level of o.O going on.


(James Cook) #11

Thanks. I will keep an eye on these logs and report anything of note. The error only seems to have happened when I’ve been interacting with the container so far. Either in the console or running the importer script


(Matt Palmer) #12

That smells like lack of memory. Does your host’s dmesg (or kernel log) report any oomkiller activity?


(James Cook) #13

You may be correct

dmesg | egrep -i “killed process”

returns

[785023.865810] Killed process 28218 (ruby) total-vm:1841856kB, anon-rss:398216kB, file-rss:0kB
[808096.474228] Killed process 26908 (ruby) total-vm:1804880kB, anon-rss:386624kB, file-rss:0kB
[808589.865843] Killed process 24691 (ruby) total-vm:1814288kB, anon-rss:374276kB, file-rss:0kB
[808589.874840] Killed process 24775 (100-sidekiq.rb*) total-vm:1814288kB, anon-rss:375152kB, file-rss:0kB
[808637.769527] Killed process 3212 (ruby) total-vm:1755040kB, anon-rss:290624kB, file-rss:0kB
[811306.837021] Killed process 12728 (convert) total-vm:205300kB, anon-rss:127676kB, file-rss:0kB
[811730.833113] Killed process 19230 (ruby) total-vm:1851664kB, anon-rss:362160kB, file-rss:0kB
[811730.837619] Killed process 19232 (ruby-timer-thr) total-vm:1851664kB, anon-rss:363728kB, file-rss:0kB
[811730.912056] Killed process 19397 (ruby) total-vm:1724988kB, anon-rss:290648kB, file-rss:0kB

I have no idea why I’d be running out of memory though. I’m running a fresh install from a DO image. 2GB of memory.


(Matt Palmer) #14

Hmm, I’d expect to see some redis-server OOMs in there, too, because that’s usually what kills RDB saves. Have you setup a swap file in your droplet? That helps the system keep going in the face of memory pressure during imports, rebuilds, and so on.


(James Cook) #15

Not in this one yet no. I’ll make it a priority and see if that helps. Thanks for your help :thumbsup:


(Sung Kim) #16

Did you figure this out?


(Sung Kim) #17

I also got many errors:
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk.
Commands that may modify the data set are disabled.

./launcher logs app shows me

41:M 24 Jan 13:29:24.012 # Can’t save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:30.024 * 1 changes in 900 seconds. Saving…
41:M 24 Jan 13:29:30.024 # Can’t save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:36.042 * 1 changes in 900 seconds. Saving…
41:M 24 Jan 13:29:36.043 # Can’t save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:42.054 * 1 changes in 900 seconds. Saving…
41:M 24 Jan 13:29:42.054 # Can’t save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:48.067 * 1 changes in 900 seconds. Saving…
41:M 24 Jan 13:29:48.067 # Can’t save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:54.082 * 1 changes in 900 seconds. Saving…
41:M 24 Jan 13:29:54.082 # Can’t save in background: fork: Cannot allocate memory

Is this a temporary? I need to restart the server to recover this error.

Any thoughts?


(Rafael dos Santos Silva) #18

What’s the output of

free -m

(Buford Taylor) #19

I’m also experiencing this issue and I’m not sure where to go from here

tail unicorn.stderr.log
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Mem

free
             total       used       free     shared    buffers     cached
Mem:       2049940    1791900     258040     210160       9532     274436
-/+ buffers/cache:    1507932     542008
Swap:            0          0          0


df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       991M  4.0K  991M   1% /dev
tmpfs                      201M  356K  200M   1% /run
/dev/disk/by-label/DOROOT   40G   15G   24G  38% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                      1001M  1.2M 1000M   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        40G   15G   24G  38% /var/lib/docker/aufs/mnt/b34e2e05c05cfc59ac267c11137745713e4d779c22dd2da8074e9d840db147f
shm                         64M  4.0K   64M   1% /var/lib/docker/containers/1e10c57a8397d72f999a9dfb5169194558110c6e24adb557229ac4b39b5ad5a/shm

Every day or so, discourse will fail and start serving 500 errors. I inherited this Discourse server and am new to it, so I’m open to reading material to point me in the right direction.

People tend to say “make a swap file.” Am I overthinking this or should I just make a swap file?


(Jeff Atwood) #20

It never hurts to have a swap file and it is recommended so I would start with that. Then rebuild from the command line.