High CPU usage after 2.2 update?

I am getting a stream of CPU usage alerts from my server, possibly after updating to Discourse 2.2.

Looking at the usage, it looks like there is a lot of image conversion stuff going on, eg:

convert png:/var/www/discourse/public/uploads/default/original/2X/9/91918d2c21632c93c696227e94fa8767c8b20449.png[0] -auto-orient -gravity center -background transparent -thumbnail 1035x1855^ -extent 1035x1855 -interpolate catrom -unsharp 2x0.5+0.7+0 -interlace none -quality 98 -profile /var/www/discourse/vendor/data/RT_sRGB.icm png:/tmp/discourse-thumbnail20190204-13233-6zk1cu.png

Is there some image conversion process that is going to finish at some point, or should I be wondering what else is going on?

Thanks.

https://meta.discourse.org/t/performance-issues-with-2-2-0-beta7/105794

2 Likes

Yes. You should reduce the rebake count setting (I can’t remember the exact name) to 20.

1 Like

it’s rebake old posts count.

You can find it at:

/admin/site_settings/category/developer?filter=rebake

It defaults to 80 and is usually fine at this value but the one-time operation introduced as a part of 2.2 does stress more resource-constrained systems.

4 Likes

Thanks, I have set that to 20, and I will see if that resolves the issue.

I take it once this one-time operation is finished the resource usage will return to normal?

You only need to lower it when you see it constantly at high CPU. It will keep peaking at 100% then going back to normal etcetera.
Lowering the count will only make the peaks less wide (i.e. shorter in time), not less high. It will also make the entire process take 4x as long.

5 Likes

And it may be that the way that it works now there is no performance hit for the high cpu. @peternlewis did you experience performance issues or just notice the high cpu? If it was the latter, then you’d likely not want to change the default.

1 Like

I noticed a constant stream of alert notifications of high CPU for more than 5 minutes. I did not notice any particular performance issues with the forum, but then I might not even if there were some - others might notice issues without me noticing anything. What I noticed was many monitor alerts of abnormally high CPU usage.

Since these alerts required high CPU usage for at least five minutes, I would presume that with the setting as it was it was spending more than five minutes out of every 15 at high CPU which seems dangerously close to continuous high CPU usage and perhaps ending up with the system starting the next batch before finishing the previous batch depending on how it is implemented, which could then lead to a cascade performance failure. Unless the 15 minute in the preference does not start counting until the end of the batch…

No it won’t end up overlapping like that. As people have pointed out in the other topic you can just make the number smaller and have it take longer. The world won’t end if the reprocessing isn’t done immediately.

No user reports is a good sign though.

1 Like

In that case, you probably didn’t need to do anything. When these changes were first made some people experienced delays in notifications because the sidekiq queue was full of images getting reprocessed, but I am pretty sure that’s no longer the case. It’s really not a problem if your computer has to work hard. It’s what we pay them for, after all.

2 Likes

I did a lot of work making this rather expensive process as seamless as possible, we have the operating system de-prioritize all the image magick convert jobs so it loses out when CPU is asked for and other processes like the DB, Redis or Forum need it.

4 Likes

That’s what I thought. I think that there was a 3-6 day window when "reduce rebake old posts count" was a good idea, but then realized that it’s no longer necessary. I just hope I can find all the places that I changed it. :slight_smile:

3 Likes