I am consistently getting this in the logs - with values between ~100k to ~1.35m - but the readings near 100k seem to be quite common: Your Redis network connection is performing extremely poorly. Last RTT readings were [97069, 103986, 98459, 100762, 381617], ideally these should be < 1000. Ensure Redis is running in the same AZ or datacenter as Sidekiq. If these values are close to 100,000, that means your Sidekiq process may be CPU-saturated; reduce your concurrency and/or see https://github.com/mperham/sidekiq/discussions/5039 This indicates that perhaps Redis isn’t able to use enough CPU? There seems to be plenty of breathing room for CPU and ram on the server itself though. also: Sidekiq is consuming too much memory (using: 3570.19M) for 'www.example.com', restarting This is using the all in one app.yml with Discourse stable 3.3.2. From the app.yml: UNICORN_SIDEKIQS: 9 DISCOURSE_SIDEKIQ_WORKERS: 5 I added this configuration to the host also: [image]MKJ's Opinionated Discourse Deployment Configuration Kernel configuration Redis (one of the key components on which Discourse is built) strongly recommends disabling transparent huge pages when using on-disk persistence (which Discourse does), and I also allow memory overcommit. echo 'sys.kernel.mm.transparent_hugepage.enabled=never' > /etc/sysctl.d/10-huge-pages.conf echo 'vm.overcommit_memory=1' > /etc/sysctl.d/90-vm_overcommit_memory.conf sysctl --system Sidekiq dashboard info: [Screenshot 2024-12-13 at 04.47.36] [Screenshot 2024-12-13 at 05.04.20] It does seem like Redis is not able to surpass 1024M memory usage. If anyone has any ideas, I’d appreciate it! :meow_heart:

To follow up with this, I’m having this same issue with Jobs::PostAlert: [image] PostAlerter job hangs/OOMs as of late Bug Ever since https://github.com/discourse/discourse/commit/9cf981f1f15d3a22929003f0ca59ac2fdd28d7d4 (presumably!) we’re having Sidekiq queues fill up/OOM with stuck PostAlert jobs: [image] [image] Given this commit has already had a similar regression (https://github.com/discourse/discourse/pull/22339) it’s fairly suspect - we’ll try downgrading to a commit before the aforementioned one now though and will report back with the results. With those jobs often going up to 15 minutes when using 4 sidekiqs with 5 (default) threads with current testing. Seems like the jobs per second speed for Sidekiq is mostly dependent on how many of those jobs are being ran simultaneously and how many threads are free for the other jobs. Increasing Sidekiqs to 6 or higher (5 threads) will increase the queue clearing speed, but postgres will crash fairly regularly (I am guessing from too many Jobs::PostAlert jobs being ran simultaneously. This is on Stable 3.3.2. The changes and fixes from the linked thread seem to be already be implemented in 3.3.2, if I am not mistaken.

Have you rebooted the server since making those kernel config changes? [image]3.1.x to 3.2.0 upgrade hangs/fails on 1GB instance my list of standard instant diagnostics: cat /etc/lsb-release uptime df -h / free vmstat 5 5 dmesg|egrep -i "memory|oom|kill" ps auxrc Maybe lscpu would also be helpful

[image] markersocial: From the app.yml: UNICORN_SIDEKIQS: 9 DISCOURSE_SIDEKIQ_WORKERS: 5 [image] You should never bump UNICORN_SIDEKIQS that high, only increasing workers but [image] markersocial: postgres will crash fairly regularly This should never happen. The possibilities are: You are constrained on resources because either a) Your site has over grown the server resources b) You are misallocating resources There is a bug somewhere in the stack I’d start making UNICORN_SIDEKIQS: 1 DISCOURSE_SIDEKIQ_WORKERS: 20 which should release some RAM from your server. For further information you will need to run the offending jobs in a PostgreSQL console and report what is the bottleneck.

Apologies for disappearing and thank you for the responses. :slight_smile: I believe that the main issue for Redis being slow, was that THP was still enabled (when I had thought otherwise): [image] MKJ's Opinionated Discourse Deployment Configuration Sysadmins I think there may be an issue with the THP configuration in the wiki. I was having issues with Redis on Ubuntu: Your Redis network connection is performing extremely poorly. Last RTT readings were [96585, 101554, 97189, 99769, 94618], ideally these should be < 1000. Ensure Redis is running in the same AZ or dat I believed that THP was already disabled (from following the wiki) - but turned out it was still enabled sweat_smile. Disabling THP ended up resolving the above for me iirc (late la… For PG crashing, the main solution for me was adding this to the app.yml: docker_args: - "--shm-size=34g" With the value set to db_shared_buffers + 2GB, with db_shared_buffers being 25% of the total host machine ram. Overriding the default 512m: launcher 4ae1f6914 if [ -z "$SKIP_MAC_ADDRESS" ] ; then mac_address="--mac-address $($docker_path run $user_args -i --rm -a stdout -a stderr $image /bin/sh -c "echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/02:\1:\2:\3:\4:\5/'")" fi if [ ! -z "$START_CMD_ONLY" ] ; then docker_path="true" fi set -x $docker_path run --shm-size=512m $links $attach_on_run $restart_policy "${env[@]}" "${labels[@]}" -h "$hostname" \ -e DOCKER_HOST_IP="$docker_ip" --name $config -t "${ports[@]}" $volumes $mac_address $user_args \ $run_image $boot_command ) exit 0 } run_run() { set_template_info launcher 4ae1f6914 } run_run() { set_template_info set_volumes set_links set_run_image unset ERR (exec $docker_path run --rm --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" -i -a stdin -a stdout -a stderr $volumes $run_image \ /bin/bash -c "$run_command") || ERR=$? if [[ $ERR > 0 ]]; then exit 1 fi } run_bootstrap() { set_template_info launcher 4ae1f6914 pups_command="$pups_command git pull && git checkout $pups_version &&" fi pups_command="$pups_command /pups/bin/pups --stdin" fi echo $pups_command declare -i BOOTSTRAP_EXITCODE rm -f $cidbootstrap echo "$input" | $docker_path run --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" --cidfile "$cidbootstrap" -i -a stdin -a stdout -a stderr $volumes $image \ /bin/bash -c "$pups_command" BOOTSTRAP_EXITCODE=$? CONTAINER_ID=$(cat "$cidbootstrap") rm -f "$cidbootstrap" # magic exit code that indicates a retry if [[ $BOOTSTRAP_EXITCODE -eq 77 ]]; then $docker_path rm "$CONTAINER_ID" exit 77

[image] markersocial: For PG crashing, the main solution for me was adding this to the app.yml: docker_args: - "--shm-size=34g" I looked back at your posting history, and I see in Very slow Sidekiq issue … massive numbers of unread user notifications that you were running a 32 core 128 GB server, with a very large and active userbase. So in that context, I see why 34G is not such a large number! For context, though, it might be helpful (and interesting) to know the size of your setup - possibly here or even in your bio? (maybe daily and monthly active users, size of database backups, server config in RAM, swap, disk, CPUs.) Maybe even a thread where we just share our stats - large and small.

"Your Redis network connection is performing extremely poorly"

Product Support Self-hosting

supermathie (Michael Brown) December 19, 2024, 4:11pm 3

Postgres should never crash and generally indicates a postgres bug or some sort of larger problem.

Do you have logs?

Topic		Replies	Views
Redis used_memory limited to 1024mb? Product Support	2	272	February 16, 2026
Could sidekiq queue be reason for 500 errors? Self-hosting server-resources	30	4396	June 13, 2018
Sidekiq is consuming too much memory, restarting Self-hosting	38	9518	October 13, 2020
Redis connection timed out Self-hosting	29	10305	July 2, 2018
Locks up while running Sidekiq after migration Self-hosting	6	2524	December 24, 2017

"Your Redis network connection is performing extremely poorly"

Related topics