Increased CPU Usage since 3.4.0.beta4-dev ( 58f75ed205 ) upgrade

Support
1

I have seen substantial increase in CPU usage since upgrading this weekend. The CPU usage of RUBY appears to be the primary driver. This was referenced by another discourse user in this topic.

As you can see from the graphs below, CPU usage and load pre-upgrade were much lower than post-upgrade. The upgrade occurred on the evening of 1/31.

The image shows graphs depicting CPU and Load percentage from 27 January to 2 February. (Captioned by AI)
The image shows graphs depicting CPU and Load percentage from 27 January to 2 February. (Captioned by AI)985×619 68.1 KB

Here are two images of TOP taken 33 hours apart:

TOP data from 2/1 @ 1:00pm CT
TOP data from 2/1 @ 1:00pm CT689×219 9.92 KB

TOP data from 2/2 @ 9:45pm CT.
TOP data from 2/2 @ 9:45pm CT.649×192 8.48 KB

In 33 hours, there is a significant ruby CPU usage. Based on the top data, I have seen 2x CPU usage in the last 33 hours over 22 days. In 33 hours, I have seen 11 hours of CPU time. (648 minutes of CPU time across 5 PIDs)

Additional Data:

  • Traffic has been down over the last two days by about 10%. (analytics and dashboard)
  • Standard single container discourse install (no-chat)
  • Sidekiq queues are minimal (1K to 2K per day)
  • Nothing seems unusual in the discourse logs
  • I run on a DO server with 8GB RAM and 2 AMD vCPUs.

This isn’t the case where the server is down critical, but servers that run at 5% to 7% are much happier than those that run at 25%.

What info can I provide to assist in troubleshooting this issue?

tia

3 Likes
2

Lets leave this in support for a bit till we determine if there is a bug.

Can you enter the container and run an htop from the inside (you will have to install it) that way you will be able to tell which specific process is consuming high amounts of cpu.

You can get a bit more visibility using a technique like this: Debugging 100% CPU usage in production Ruby on Rails systems

Most likely though, is sidekiq /sidekiq somehow overloaded on your instance. (I would look a schedular particularly)

3

htop views.

Here is a 30 second video:

Random Screen shots:

The image displays a Unix-like system’s process list or task manager showing various running processes, with a focus on multiple instances of 'unicorn worker' processes, and provides specific details such as user ID, process ID, priority, CPU, memory usage, and estimated runtime. (Captioned by AI)
The image displays a Unix-like system’s process list or task manager showing various running processes, with a focus on multiple instances of 'unicorn worker' processes, and provides specific details such as user ID, process ID, priority, CPU, memory usage, and estimated runtime. (Captioned by AI)1140×852 98.8 KB

This image displays a Unix-like system's process and CPU usage output, featuring multiple worker processes associated with Chrome, javascript, and Redis, along with details such as PID, user, and percent CPU usage. (Captioned by AI)
This image displays a Unix-like system's process and CPU usage output, featuring multiple worker processes associated with Chrome, javascript, and Redis, along with details such as PID, user, and percent CPU usage. (Captioned by AI)1139×860 99.7 KB

This image is a screenshot of a Unix system's process list showing various processes, primarily related to a Redis server, with details such as PID, user, priority, VIRT, and other system-specific information. (Captioned by AI)
This image is a screenshot of a Unix system's process list showing various processes, primarily related to a Redis server, with details such as PID, user, priority, VIRT, and other system-specific information. (Captioned by AI)1153×878 101 KB

This image is a screenshot of a Unix-like operating system command-line terminal displaying a running "discourse" process output with various process details, system information, and commands executed. (Captioned by AI)
This image is a screenshot of a Unix-like operating system command-line terminal displaying a running "discourse" process output with various process details, system information, and commands executed. (Captioned by AI)1134×505 59.4 KB

Tree View:

The image shows a section of server logs with timestamps, commands, and user actions, alongside repeated lines in green and red text indicating processes such as "sdekiq" and "unicorn worker" interacting with "discourse" and related configuration commands. (Captioned by AI)
The image shows a section of server logs with timestamps, commands, and user actions, alongside repeated lines in green and red text indicating processes such as "sdekiq" and "unicorn worker" interacting with "discourse" and related configuration commands. (Captioned by AI)1151×674 104 KB

The image shows a screenshot of a Linux system using the top command, displaying processes and system metrics. (Captioned by AI)
The image shows a screenshot of a Linux system using the top command, displaying processes and system metrics. (Captioned by AI)1150×879 119 KB

sidekiq:

This image is a screenshot of a queue management dashboard displaying a processed count, a few failed images, and queues with detailed information like busyness, enqueued, retries, scheduled, and dead images, along with a delete action for each queue. (Captioned by AI)
This image is a screenshot of a queue management dashboard displaying a processed count, a few failed images, and queues with detailed information like busyness, enqueued, retries, scheduled, and dead images, along with a delete action for each queue. (Captioned by AI)1189×428 43.8 KB

The line chart shows the progression of an image captioning task over several days, marked by vertical markers indicating failed attempts at different times. (Captioned by AI)
The line chart shows the progression of an image captioning task over several days, marked by vertical markers indicating failed attempts at different times. (Captioned by AI)1203×296 50.5 KB

This image shows a server panel for a process named "discourse-app:817" running with 0% utilization, 461 MB of RSS, and it can be paused or stopped using the provided buttons. (Captioned by AI)
This image shows a server panel for a process named "discourse-app:817" running with 0% utilization, 461 MB of RSS, and it can be paused or stopped using the provided buttons. (Captioned by AI)1229×537 73.1 KB

Let me know if there is something specific you need to see. I

2 Likes
4

Yeah something is off:

top -H -p PID_OF_UNICORN

A screenshot showing a top command output indicating system resources and processes, with 367 threads, predominantly running a ruby or colourer-2480 process, and a mostly free but used RAM usage. (Captioned by AI)
A screenshot showing a top command output indicating system resources and processes, with 367 threads, predominantly running a ruby or colourer-2480 process, and a mostly free but used RAM usage. (Captioned by AI)1242×610 102 KB

I suspect you will see V8 DefaultWorker there, I think this is a regressing in mini_racer… will revert it to see if it resolves this.

1 Like
5

OK this is reverted now,

Let me know if the commit restores performance.

6 Likes
6

Yes, it resolved the high CPU issue. My 1-minute and 5-minute load is about 1/3 of the previous values. That is with htop and netdata now running on the system.

HTOP video

Do Graph

The image shows two line graphs depicting CPU percentage and load (1/5/15) over various time intervals, with different lines representing different loads (0.100, 1.000, 15.000, and 50.150). (Captioned by AI)
The image shows two line graphs depicting CPU percentage and load (1/5/15) over various time intervals, with different lines representing different loads (0.100, 1.000, 15.000, and 50.150). (Captioned by AI)988×553 28.2 KB

I would expect the CPU usage and load to slowly decrease as more database queries are cached in the system.

Load Table:

time Pre-fix post-fix
1 min 0.4 to 0.6 0.06 to 0.1
5 min 0.39 to 0.5 0.15 to 0.18

The 15-minute metric is impacted by a rebuild. I will post some more stats later this morning.

Thank you for the late-night fix.

3 Likes
7

Sorry about this, the mini_racer upgrade has been a big adventure. Hopefully we get through it soon.

3 Likes
8

Thank you for the fast response to investigate.

I am sure you had other things planned for the day versus a rollback.

As a new Discourse user, 2 weeks since migration, the product has been great to work with.

2 Likes
9

Similar story here aswell.

The image displays a detailed graph with two charts: the upper chart is a colorful line graph showing CPU usage over time, highlighted with a significant peak on January 27th, marked with CPU usage of 2.03%, alongside the lower chart indicating system load in a line graph format, featuring negligible... (Captioned by AI)
The image displays a detailed graph with two charts: the upper chart is a colorful line graph showing CPU usage over time, highlighted with a significant peak on January 27th, marked with CPU usage of 2.03%, alongside the lower chart indicating system load in a line graph format, featuring negligible... (Captioned by AI)2208×1234 295 KB

[Edit: seems to be fixed now after updating to latest branch]

10

Here is a performance review 18 hours after the rebuild. The load table says it all.

Load Table:

time Pre-fix post-fix
1 min 0.4 to 0.6 0.03 to 0.05
5 min 0.39 to 0.5 0.09
15 min 0.68 0.12

Legend:

  • Red arrow - rebuilt the app
  • Purple arrow - turned off netdata

The image shows two graphs: one with a red arrow indicating high CPU usage and another with a purple arrow showing spikes in load, both occurring at different times during the day. (Captioned by AI)
The image shows two graphs: one with a red arrow indicating high CPU usage and another with a purple arrow showing spikes in load, both occurring at different times during the day. (Captioned by AI)984×551 54.8 KB

11

Note, closing the loop, the bug causing it was this:

I updated the gem. One immediate advantage is that it appears this version of v8 uses slightly less memory which is nice.

1 Like