CPU usage increases steadily

Support
1

I see this Sidekiq CPU usage pattern (the growing part is Sideqik):

This image displays a graph showing the CPU usage percentages of various applications over time, with lines indicating the usage for "apps.plugin," "auditd," "backup," "bash," "boot," and "build," on a computer screen. (Captioned by AI)
This image displays a graph showing the CPU usage percentages of various applications over time, with lines indicating the usage for "apps.plugin," "auditd," "backup," "bash," "boot," and "build," on a computer screen. (Captioned by AI)839×356 58.8 KB

Maybe connected with Increased CPU Usage since 3.4.0.beta4-dev ( 58f75ed205 ) upgrade?

How should I investigate?

2 Likes
3

when did you rebuild your container last? if it was a while ago I recommend you rebuild now.

4

The first usage drop was after the recommended rebuild.
Last rebuild is one hour old but with same usage pattern.

5

have you looked at sidekiq, is there a backlog? visit /sidekiq

6

The image shows a summary panel with a counter of 37 job applications, 5 candidates interviewed, and various zeroes across different categories, including rejected applicants, candidates in the applicant pool, and job openings. (Captioned by AI)
The image shows a summary panel with a counter of 37 job applications, 5 candidates interviewed, and various zeroes across different categories, including rejected applicants, candidates in the applicant pool, and job openings. (Captioned by AI)1826×138 18.9 KB

7 
./launcher enter app
apt-get update && apt-get install -y htop
htop

Is CPU high at the moment?

If it is high, then try to inspect CPU on a process that is running high using:

ps 
top -H -p PID_OF_HIGH_CPU
8

The image displays a Unix system running on Mindowski, showing various running processes, system status, and memory usage. (Captioned by AI)
The image displays a Unix system running on Mindowski, showing various running processes, system status, and memory usage. (Captioned by AI)1487×1076 79 KB

9

that is not showing any concerning CPU amount, CPU is hovering on zero now across all processes?

10

There is no longer running process taking much time.

HTOP shows ever changing CPU utilisation.

Netplan’s stats attribute the raising CPU utilisation to sideqik:

The image shows a graphical audio editing interface with various waveforms and tracks displayed, along with a highlighted section indicating five percent progress. (Captioned by AI)
The image shows a graphical audio editing interface with various waveforms and tracks displayed, along with a highlighted section indicating five percent progress. (Captioned by AI)458×467 48.2 KB

Is there any profiling tooling for sideqik I could enable?

11

the key is that you need to catch stuff misbahaving, finding background radiation is super hard.

12

Reading app/jobs/base.rb and MethodProfiler, I would like to learn if the collected metrics could be saved to a report in an accumulated way (as sums of time spent in different jobs) in a production environment.

I understand that detailed logging can be enabled via ENV["DISCOURSE_LOG_SIDEKIQ"], but I expect this to generate a large amount of data in a production environment.

13

For two-container do you mean the web container or the database/redis container?

14

the cpu usage is in the web container, but I would rebuild both just in case.

15

Sorry for the hassle. The staging site was the culprit. Maildev was not running, so all mails failed. I should sleep more.

4 Likes
16

I wrote here about what I learned:

2 Likes