Hi. I just a ran a big import job on the server (Docker standard install, updated 2 weeks ago) that I am getting ready for production.
The import went well up to the point where it was updating reply counts, then it failed.
Now I get high CPU usage in my server, with exhausted 2GB swap:
free -m
total used free shared buff/cache available
Mem: 2000 1620 71 251 308 13
Swap: 2047 2047 0
I’ve read a few related threads on these forums, but they seem to mention old memory leaks that are supposed to be fixed now.
I have restarted the container, but it went back to the same state… top shows redis-server eating up CPU.
Some other outputs:
pedro@discourse:~$ dmesg | egrep -i "killed process"
[267432.830391] Killed process 26850 (redis-server) total-vm:2003500kB, anon-rss:784556kB, file-rss:0kB
[267448.176311] Killed process 26872 (redis-server) total-vm:2003500kB, anon-rss:792068kB, file-rss:0kB
[267482.515811] Killed process 26884 (redis-server) total-vm:2003500kB, anon-rss:793852kB, file-rss:0kB
[267551.301553] Killed process 26910 (redis-server) total-vm:2003500kB, anon-rss:793084kB, file-rss:0kB
[267668.666869] Killed process 27151 (redis-server) total-vm:2003500kB, anon-rss:701624kB, file-rss:4kB
[267691.245445] Killed process 27289 (redis-server) total-vm:2003500kB, anon-rss:738988kB, file-rss:0kB
[267706.403592] Killed process 27311 (redis-server) total-vm:2003500kB, anon-rss:747404kB, file-rss:0kB
[267725.563074] Killed process 27326 (redis-server) total-vm:2003500kB, anon-rss:782692kB, file-rss:0kB
[267743.125236] Killed process 27345 (redis-server) total-vm:2003500kB, anon-rss:747368kB, file-rss:0kB
[267759.046725] Killed process 27362 (redis-server) total-vm:2003500kB, anon-rss:773652kB, file-rss:0kB
[267785.725355] Killed process 27377 (redis-server) total-vm:2003500kB, anon-rss:778980kB, file-rss:0kB
[267806.894015] Killed process 27401 (redis-server) total-vm:2003500kB, anon-rss:780564kB, file-rss:100kB
[267837.641000] Killed process 27416 (redis-server) total-vm:2003500kB, anon-rss:783924kB, file-rss:0kB
[267859.241967] Killed process 27441 (redis-server) total-vm:2003500kB, anon-rss:780548kB, file-rss:0kB
[267871.387644] Killed process 27460 (redis-server) total-vm:2003500kB, anon-rss:674692kB, file-rss:0kB
[267889.559699] Killed process 27471 (redis-server) total-vm:2003500kB, anon-rss:735068kB, file-rss:0kB
[267907.637106] Killed process 27489 (redis-server) total-vm:2003500kB, anon-rss:775688kB, file-rss:0kB
[267926.757203] Killed process 27505 (redis-server) total-vm:2003500kB, anon-rss:782580kB, file-rss:0kB
[267931.645424] Killed process 27520 (redis-server) total-vm:2003496kB, anon-rss:198056kB, file-rss:0kB
pedro@discourse:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 991M 0 991M 0% /dev
tmpfs 201M 21M 180M 11% /run
/dev/vda1 58G 29G 30G 50% /
tmpfs 1001M 0 1001M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1001M 0 1001M 0% /sys/fs/cgroup
/dev/vda15 105M 3.4M 101M 4% /boot/efi
tmpfs 201M 0 201M 0% /run/user/1002
I also have a full log of the importer output on a file, it starts throwing these very frequently from a certain point on:
66564 / 78241 ( 85.1%) [413 items/min] Exception while creating post 68204. Skipping.
MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.0.1/lib/redis/client.rb:119:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.0.1/lib/redis.rb:523:in `block in del'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.0.1/lib/redis.rb:45:in `block in synchronize'
/usr/local/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.0.1/lib/redis.rb:45:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.0.1/lib/redis.rb:522:in `del'
/var/www/discourse/lib/discourse_redis.rb:216:in `block in del'
/var/www/discourse/lib/discourse_redis.rb:169:in `ignore_readonly'
/var/www/discourse/lib/discourse_redis.rb:214:in `del'
/var/www/discourse/lib/distributed_mutex.rb:37:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/var/www/discourse/lib/post_creator.rb:333:in `transaction'
/var/www/discourse/lib/post_creator.rb:165:in `create'
/var/www/discourse/script/import_scripts/base.rb:566:in `create_post'
/var/www/discourse/script/import_scripts/base.rb:514:in `block in create_posts'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.1/lib/patches/db/mysql2.rb:8:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.1/lib/patches/db/mysql2.rb:8:in `each'
/var/www/discourse/script/import_scripts/base.rb:501:in `create_posts'
script/import_scripts/kunenapgr.rb:150:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:884:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:883:in `loop'
/var/www/discourse/script/import_scripts/base.rb:883:in `batches'
script/import_scripts/kunenapgr.rb:128:in `import_posts'
script/import_scripts/kunenapgr.rb:78:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
script/import_scripts/kunenapgr.rb:186:in `<main>'
Can anybody please help me troubleshoot? What more information do you need?