Performance issues with 2.2.0.beta7

I’ve noticed there’s been a couple of major shifts in performance ever since I updated to this.

There is a noticeable slowdown in the forum performance and the Sidekiq has gone from a steady 0 to a rash of failures (which required a rebuild to get some of them to eventually go through) and having intermittent errors that are fixed with automatic retries:

Every time I have caught a failure it’s said the database is in recovery mode.

It’s hosted on Dreamcompute Lightspeed (2 VCPU, 4 gig RAM) and has previous handled very well and traffic on the site has diminished greatly since November 2018 - so basically I’m wondering if there’s any known issues that his upgrade could be causing with configurations?

Are you on tests-passed or beta branch?

The default: tests-passed

Largely because I did just leave it on default.

That is fine then. I strongly recommend you update to latest again, cause I did tweak this on Friday again.

Expect slightly higher CPU usage for a few days while we re-optimize some old images, but it should return to normal soonish.

See:

and

13 Likes

I also just updated this morning using the admin update utility.

I upgraded my server but its still bogging it down hard. How long does this take?

You can reduce the impact by reducing the rebake old posts count to a smaller number, maybe 20 … that is how many rebakes we do in 15 minutes.

Highly depends on the forum how long this impact will be in place.

3 Likes

Thanks set it lower. Defaulted to 80. Was running on t2.small in aws. upgraded to t3.medium but was still keeping both cores locked at 100% site has been on and off 500ing all day.

1 Like

Perhaps this default value should be lowered? It is pretty concerning to me that this basically crashed my forum all day. I planned an update on Sunday after I got the update notice email. I don’t feel the release notes conveyed that this would happen either.

1 Like

Release notes are not relevant to you as you pulled latest, not the beta release. You got more changes in latest.

That is true. I assume update just does a git pull. I did not know these changes were not apart of the release. I just usually do it the Sunday after the email goes out.

I would still encourage a lower default value on the rebake as forcing a rebake crippled my small instance.

3 Likes

Hi everyone,

Ours is doing the same :+1:t2:

Anyone know if there a query I can run periodically with the Data Explorer plugin to see how far through the task I am? :thinking:

Be sure to pull latest and do a full rebuild as @sam committed some crucial optimizations today for this.

1 Like

Thanks @codinghorror I just updated to v2.2.0.beta7 +83 this morning (approx 2hrs ago - 09:30GMT)

I updated via the /admin/upgrade/ page.

Should I rebuild the app from the console too?

Not required, but would not hurt anything

6 Likes

I’ve moved the default rebake old posts count from 80 down to 50 as it was slowing our site right down too.

Just to repose my earlier question:

Or should I simply base this on 50x posts every 15 mins = 200 posts per hour.

47,000 posts @ 200 per hour = 235 hours (or 9.7 days)

Hmmm :thinking:

Well I updated this morning to 2.2.0.beta7 +84

and the server is picking CPU for about 5 hours now … and we have only 2.6K topics and 17K posts total…

Maybe something else is going on?

Images also temporary disappear regularly from topics and after some time when regenerated new images are ready become visible…

Did it help lower the CPU?

Yeah we’re still running pretty much flat out too:

And this is when I dropped from 80 rebake posts per 15 mins, down to 50x posts:

image

So I guess each peak is the point at which it does the next batch.

9 days at this rate though? :confused:

Similar, it was so good before :slight_smile:

1 Like

Same here… 30 day graphs :cry: