I just rebuilt a site with 8GB ram. Apparently sidekiq had died a few days ago and is now trying to catch up on 500K queued processes. It runs for a while and then redis stops responding. Logs have errors like these: Error connecting to Redis on localhost:6379 (Redis::TimeoutError) subscribe failed, reconnecting in 1 s econd. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis/client.rb:34 5:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/l ib/redis/client.rb:331:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/re dis-3.3.5/lib/redis/client.rb:101:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/ gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2 .4.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.4 .0/gems/redis-3.3.5/lib/redis/client.rb:276:in `with_socket_timeout'", "/var/www/discourse/vendor/bundl e/ruby/2.4.0/gems/redis-3.3.5/lib/redis/client.rb:133:in `call_loop'", "/var/www/discourse/vendor/bundl e/ruby/2.4.0/gems/redis-3.3.5/lib/redis/subscribe.rb:43:in `subscription'", "/var/www/discourse/vendor/ bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis/subscribe.rb:12:in `subscribe'", "/var/www/discourse/vendo r/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2775:in `_subscription'", "/var/www/discourse/vendor/ bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2143:in `block in subscribe'", "/var/www/discourse/vend or/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'", "/usr/local/lib/ruby/ 2.4.0/monitor.rb:214:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3 .5/lib/redis.rb:58:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib /redis.rb:2142:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib /message_bus/backends/redis.rb:336:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.4.0 /gems/message_bus-2.1.1/lib/message_bus.rb:525:in `global_subscribe_thread'", "/var/www/discourse/vendo r/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus.rb:473:in `block in new_subscriber_thread'"] Job exception: Connection timed out Job exception: Connection timed out Job exception: Error connecting to Redis on localhost:6379 (Redis::TimeoutError) Job exception: Connection timed out Job exception: Connection timed out I’ve got this set now: db_shared_buffers: "2048MB" db_work_mem: "50MB" I’m going to try doubling db_shared_buffers next. Edit: but that didn’t help either. My multisite instance has also been having the database die several times a day. I thought it was coincident with backups, but I’m not convinced that’s true. Has something changed about redis in the past couple weeks that could affect this?

After further investigation, I’ve confirmed a bug in Docker, and it would be entirely capable of causing all of the seemingly unrelated symptoms you described. The linked post has more details and a rundown of how to downgrade Docker, which I’m pretty confident will solve all these problems.

[image] pfaffman: I’m going to try doubling db_shared_buffers next. Edit: but that didn’t help either. Those are for PostgreSQL, not redis. Is this site high traffic? What is the value of redis size and ops per second during load? I found a self hosted site failing hard to keep up because of redis persistence process, and tweaking it made everything smooth again.

Thanks, @Falco. That sounds like what I need to fix, but I know not of what you speak. It’s about 15K page views/day, but right now there are a 400K sidekiq jobs clogging things up. Uh, where do I find redis size and ops per second?

Linux 4.4.0-78-generic (site.com) 12/18/2017 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 18.90 0.01 6.54 0.13 1.11 73.31 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 24.24 140.35 1943.41 132914772 1840484564

Could you please try the below command for 10 seconds and then share the output. iostat 1 We should be looking at the tps column

thanks, @nixie! avg-cpu: %user %nice %system %iowait %steal %idle 44.00 0.00 6.50 0.25 0.50 48.75 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 28.00 16.00 248.00 16 248 avg-cpu: %user %nice %system %iowait %steal %idle 34.94 0.00 4.81 0.25 0.76 59.24 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 27.00 16.00 316.00 16 316 avg-cpu: %user %nice %system %iowait %steal %idle 31.14 0.00 4.81 0.25 0.51 63.29 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 22.00 24.00 256.00 24 256 avg-cpu: %user %nice %system %iowait %steal %idle 32.66 0.00 5.32 0.00 0.25 61.77 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 36.00 104.00 264.00 104 264 avg-cpu: %user %nice %system %iowait %steal %idle 42.32 0.00 6.55 0.25 0.50 50.38 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 22.00 8.00 196.00 8 196 avg-cpu: %user %nice %system %iowait %steal %idle 32.33 0.00 4.76 0.25 0.50 62.16 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 22.00 24.00 208.00 24 208 avg-cpu: %user %nice %system %iowait %steal %idle 35.19 0.00 6.84 0.00 1.01 56.96 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 19.00 16.00 192.00 16 192 avg-cpu: %user %nice %system %iowait %steal %idle 34.78 0.00 3.58 0.00 0.00 61.64 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 28.00 48.00 208.00 48 208 avg-cpu: %user %nice %system %iowait %steal %idle 41.94 0.00 5.37 0.77 0.51 51.41 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 70.00 464.00 220.00 464 220 avg-cpu: %user %nice %system %iowait %steal %idle 51.75 0.00 9.50 0.25 1.00 37.50 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 53.47 229.70 150.50 232 152 avg-cpu: %user %nice %system %iowait %steal %idle 38.01 0.00 6.38 2.55 1.53 51.53 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 26.26 64.65 185.86 64 184 avg-cpu: %user %nice %system %iowait %steal %idle 36.00 0.00 6.75 0.00 2.75 54.50 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 19.80 23.76 312.87 24 316 avg-cpu: %user %nice %system %iowait %steal %idle 42.24 0.00 6.36 0.00 1.53 49.87 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 10.00 0.00 104.00 0 104 avg-cpu: %user %nice %system %iowait %steal %idle 33.17 0.00 5.45 0.25 1.98 59.16 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 19.00 0.00 412.00 0 412 avg-cpu: %user %nice %system %iowait %steal %idle 31.47 0.00 6.09 0.25 0.51 61.68 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 35.00 8.00 304.00 8 304 Sometimes it’s >200 and a few times it’s >2000.

Disk IOPS aren’t going to be relevant to Redis’ operation, because it’s an in-memory data store. What Falco’s asking for is Redis operations per second. From memory, redis-cli info stats from inside the container should do the trick.

# Stats total_connections_received:56 total_commands_processed:433486 instantaneous_ops_per_sec:837 total_net_input_bytes:34247556 total_net_output_bytes:25502898 instantaneous_input_kbps:59.92 instantaneous_output_kbps:8.48 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:518 evicted_keys:0 keyspace_hits:265122 keyspace_misses:94064 pubsub_channels:1 pubsub_patterns:0 latest_fork_usec:9114 migrate_cached_sockets:0 Looks like it’s hanging in the 300-600 range, occasionally going over 1000. And the watched pot syndrome is in effect. This is the longest the site has stayed up.

[image] mpalmer: Disk IOPS aren’t going to be relevant to Redis’ operation Could be, if large amount of key-value churn is causing frequent dumps of the entire Redis datastore to disk.

[image] codinghorror: Could be, if large amount of key-value churn is causing frequent dumps of the entire Redis datastore to disk. Nope, because dumping the DB happens in a separate process from serving requests. Only CPU starvation and memory pressure (swapping) can cause Redis to slow down. Neither of those appear to be the case, based on the iostat data provided (CPU 50+% idle, IOPS fairly low). [image] pfaffman: instantaneous_ops_per_sec:837 That’s a respectable rate of operations, but it shouldn’t be enough to cause Redis is collapse under its own weight. Are you collecting all this data while the system is misbehaving, or during correct operation? Stats while the system is running properly aren’t as likely to show problems.

Redis connection timed out

Product Support Self-hosting

nixie December 18, 2017, 10:34pm 4

OPS per second can be checked using the iostat command.
There is very less chance that it will be installed by default.
So you might have to install it first.

Topic		Replies	Views
Discourse web interface becomes unresponsive a few minutes after starting Self-hosting	34	7358	January 8, 2018
Issue with new update: debugging options Self-hosting	16	633	August 30, 2024
Error connecting to Redis Self-hosting	17	7467	March 21, 2023
Could sidekiq queue be reason for 500 errors? Self-hosting server-resources	30	4420	June 13, 2018
Redis Error after upgrade Self-hosting	13	1898	June 3, 2020

Redis connection timed out

Related topics