CPU usage increases steadily

Product Support

AI-generated summary

The discussion revolves around steadily increasing CPU usage in a Discourse instance, primarily attributed to Sidekiq. thoka shares graphs showing the CPU usage pattern and suspects a connection to a recent upgrade. sam suggests rebuilding the container, which thoka confirms was done recently without resolving the issue. They explore potential causes, including job backlogs, with thoka providing a screenshot of Sidekiq’s job status. sam guides troubleshooting via htop and ps, but thoka notes CPU usage fluctuates and is not consistently high. thoka points to Netdata’s stats showing Sidekiq as the source. sam emphasizes the difficulty of detecting intermittent issues. thoka inquires about profiling Sidekiq jobs and saving metrics. pfaffman clarifies the container rebuild scope, and sam confirms it’s the web container. Ultimately, thoka discovers the issue was on a staging site due to a failed mail daemon, resolving the concern. thoka shares a follow-up post detailing their findings.

1

I see this Sidekiq CPU usage pattern (the growing part is Sideqik):

This image displays a graph showing the CPU usage percentages of various applications over time, with lines indicating the usage for "apps.plugin," "auditd," "backup," "bash," "boot," and "build," on a computer screen. (Captioned by AI)
This image displays a graph showing the CPU usage percentages of various applications over time, with lines indicating the usage for "apps.plugin," "auditd," "backup," "bash," "boot," and "build," on a computer screen. (Captioned by AI)839×356 58.8 KB

Maybe connected with Increased CPU Usage since 3.4.0.beta4-dev ( 58f75ed205 ) upgrade?

How should I investigate?

3

when did you rebuild your container last? if it was a while ago I recommend you rebuild now.

4

The first usage drop was after the recommended rebuild.
Last rebuild is one hour old but with same usage pattern.

5

have you looked at sidekiq, is there a backlog? visit /sidekiq

6

The image shows a summary panel with a counter of 37 job applications, 5 candidates interviewed, and various zeroes across different categories, including rejected applicants, candidates in the applicant pool, and job openings. (Captioned by AI)
The image shows a summary panel with a counter of 37 job applications, 5 candidates interviewed, and various zeroes across different categories, including rejected applicants, candidates in the applicant pool, and job openings. (Captioned by AI)1826×138 18.9 KB

7 
./launcher enter app
apt-get update && apt-get install -y htop
htop

Is CPU high at the moment?

If it is high, then try to inspect CPU on a process that is running high using:

ps 
top -H -p PID_OF_HIGH_CPU
8

The image displays a Unix system running on Mindowski, showing various running processes, system status, and memory usage. (Captioned by AI)
The image displays a Unix system running on Mindowski, showing various running processes, system status, and memory usage. (Captioned by AI)1487×1076 79 KB

9

that is not showing any concerning CPU amount, CPU is hovering on zero now across all processes?

10

There is no longer running process taking much time.

HTOP shows ever changing CPU utilisation.

Netplan’s stats attribute the raising CPU utilisation to sideqik:

The image shows a graphical audio editing interface with various waveforms and tracks displayed, along with a highlighted section indicating five percent progress. (Captioned by AI)
The image shows a graphical audio editing interface with various waveforms and tracks displayed, along with a highlighted section indicating five percent progress. (Captioned by AI)458×467 48.2 KB

Is there any profiling tooling for sideqik I could enable?

11

the key is that you need to catch stuff misbahaving, finding background radiation is super hard.

12

Reading app/jobs/base.rb and MethodProfiler, I would like to learn if the collected metrics could be saved to a report in an accumulated way (as sums of time spent in different jobs) in a production environment.

I understand that detailed logging can be enabled via ENV["DISCOURSE_LOG_SIDEKIQ"], but I expect this to generate a large amount of data in a production environment.

13

For two-container do you mean the web container or the database/redis container?

14

the cpu usage is in the web container, but I would rebuild both just in case.

15

Sorry for the hassle. The staging site was the culprit. Maildev was not running, so all mails failed. I should sleep more.

16

I wrote here about what I learned: