# Strange CPU usage since latest upgrade
**URL:** https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [April 8, 2019, 2:19pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760 "2019-04-08T14:19:22Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 2:19pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/1 "2019-04-08T14:19:22Z")
Ever since the latest update to v2.3.0.beta6+148, CPU usage has went up by around 30%. No idea what’s causing it.
Network traffic:
Disk activity:
Any ideas?
Quick puzzle: When did I upgrade?
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 2:33pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/2 "2019-04-08T14:33:37Z")
More detailed CPU data:
Seems to happen exactly 15min after the hour every hour, for almost exactly 20min.
EDIT: The peak CPU usage periods coincide with high disk READ activity (gigabytes of disk read).
So:
(1) Something is reading GB’s off the disk every hour
(2) Something is doing a lot of processing every hour
(3) For around 15min.
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 2:49pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/3 "2019-04-08T14:49:00Z")
When I did a `glances` during high CPU activity, the process shows 98% CPU with `main: discourse discourse`.
However, when I did a `top`, it shows `postmaster` with user `lxd`.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 8, 2019, 2:49pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/4 "2019-04-08T14:49:26Z")
There was an upgrade a while back that caused all images to be re-processed. That’s the likely cause.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [April 8, 2019, 2:50pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/5 "2019-04-08T14:50:27Z")
You can visit `/sidekiq/scheduler/history` and see which jobs cover the same area of the CPU spikes.
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 2:51pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/6 "2019-04-08T14:51:19Z")
Could be!
I guess I"ll wait for a few more days to see…
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 2:53pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/7 "2019-04-08T14:53:26Z")
> [@Falco](#):
>
> `/sidekiq/scheduler/history`
The only thing that is hundreds of ms is `Jobs::PeriodicalUpdates`, and that’s only 400-500ms. Everything else is \<100ms.
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 3:04pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/8 "2019-04-08T15:04:49Z")
Well, I get it. During the _exact_ time of the CPU spike, only one Sidekiq task is active, `Jobs::CleanUpUploads`.
And the _next_ task does not appear until _exactly_ at end of the CPU spike.
And the `duration` field in the history is blank, probably overflew the number field?
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 8, 2019, 3:18pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/9 "2019-04-08T15:18:40Z")
Can this be running on an infinite loop?
(1) Seems like `Jobs::CleanUpUploads` is the culprit
(2) it runs for 20min. non-stop, holding off all other Sidekiq tasks
(3) It reads 1-2GB worth of data from the disk
(4) It doesn’t write much data to the disk
(5) It does NOT incur any network traffic (all my uploads are stored in an Azure Blob storage)
(6) It keeps running for 20min. EVERY SINGLE HOUR. I don’t have that many uploads.
It almost _feels_ like the task is reading the list of uploads from the database, decides that all of them requires processing, then tries to process each upload one by one, only to fail every time because it can’t find the file(s) on the disk. One hour later, repeat.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [April 8, 2019, 3:23pm UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/10 "2019-04-08T15:23:21Z")
> [@schungx](#):
>
> all my uploads are stored in an Azure Blob storage
Oh something in the plugin can trigger an odd code path here.
>
As we are not running this plugin in production, it’s not as rock solid as our S3 code.
I recommend paying attention to the list of server processes when this is happening. It should be either a PostgreSQL query or it trying to find the non-local uploads in the disk.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [April 9, 2019, 2:14am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/11 "2019-04-09T02:14:47Z")
This is something @tgxworld should maybe have a peek at if it’s an Azure edge case?
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [April 9, 2019, 2:26am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/12 "2019-04-09T02:26:34Z")
> [@schungx](#):
>
> It almost _feels_ like the task is reading the list of uploads from the database, decides that all of them requires processing, then tries to process each upload one by one, only to fail every time because it can’t find the file(s) on the disk. One hour later, repeat.
Hmm the clean up uploads job doesn’t attempt to process any uploads since it only deletes orphaned uploads. Can you help me to run the following manually to see if it triggers the same spike?
```plaintext
cd /var/discourse
./launcher enter app
rails c
Jobs::CleanUpUploads.new.execute({})
```
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 10, 2019, 10:59am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/13 "2019-04-10T10:59:32Z")
Yes, it runs for a _long_ time occupying CPU:
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [April 11, 2019, 5:05am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/14 "2019-04-11T05:05:46Z")
Strangely it looks like it magically sorted itself out… @schungx should we close this?
---
### Author:  [@schungx](https://meta.discourse.org/u/schungx)
#### Post date: [April 11, 2019, 5:13am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/15 "2019-04-11T05:13:14Z")
OK!
Yeah, just like _MAGIC_!
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [April 11, 2019, 5:24am UTC](https://meta.discourse.org/t/strange-cpu-usage-since-latest-upgrade/113760/16 "2019-04-11T05:24:12Z")