Analyze CPU Time Spent in Background Jobs (Sidekiq)

Community wiki Sysadmins
1

You can enable Sidekiq logging to collect measurements about time spent in each job via ENV["DISCOURSE_LOG_SIDEKIQ"].

One way to enable it without rebuilding the container is to edit `/src/config/environments/production.rb` and add this code somewhere early: 
ENV["DISCOURSE_LOG_SIDEKIQ"] = "1"

Then, restart the container.

The job machinery will write measurements to log/sidekiq.log as lines of JSON, which can easily be summarized via jq.

To get the total and average time spent in jobs taking more than 5% of the total time:

sudo apt update && apt install jq -y

jq -s '
  group_by(.job_name)
  | map({
      job_name: .[0].job_name,
      count: length,
      total_duration: (map(.duration) | add),
      average_duration: (map(.duration) | add / length)
    })
  | (map(.total_duration) | add) as $grand_total
  | map(. + { percentage: (.total_duration / $grand_total * 100) })
  | map(select(.percentage > 5))
  | sort_by(.total_duration)
' log/sidekiq.log
2 Likes