Ember Performance 1.7.1 -> 1.8.1

performance

(Robin Ward) #1

This morning we upgraded Discourse from Ember 1.7.1 to Ember 1.8.1. The best news involved here is it means that there are no longer metamorphs in our code! Also, we are on track to support the future upgrades to Ember including HTMLBars, which is close to landing in beta.

Unfortunately, there are some performance regressions in Ember 1.8.1. The Ember core team is aware of them and are working on reducing them as time goes on. Many are the result of backwards compatibility between HTMLBars and Handlebars.

This post is meant to catalog some of the different performance characteristics, using the ember performance suite that we created.

The following are preliminary results; we’re still going to tune the ember performance suite and eventually provide better statistics:

overall (all platforms):

render list: 236ms -> 243ms (3% slower)
render unbound: 143ms -> 205ms (31% slower)
render complex: 602ms -> 715ms (18% slower)

chrome (desktop):

render list: 369ms -> 316ms (16% faster)
render unbound: 217ms -> 298ms (37% slower)
render complex: 647ms -> 681ms (5% slower)

chrome (android):

render list: 376ms -> 434ms (15% slower)
render unbound: 192ms -> 191ms (no diff)
render complex: 1440ms -> 1725ms (16% slower)

safari (desktop):

render list: 21ms -> 24ms (14% slower)
render unbound: 33ms -> 37ms (11% slower)
render complex: 120ms -> 143ms (19% slower)

safari (mobile):

render list: 136ms -> 189ms (29% slower)
render unbound: 73ms -> 97ms (32% slower)
render complex: 403ms -> 509ms (20% slower)

(Jeff Atwood) #2

Just as a warning, the benchmark will show a blank page in Firefox while running (we will fix) but it does work if you wait until the end.


(Kane York) #3

This was reverted earlier today, as it caused a number of bugs in the live site(s).


(Robin Ward) #4

We’re going to unrevert today and figure them out. A few of them popped up just as I was heading out to a Christmas show with my wife and couldn’t fix remotely. We’ll get it all patched up today!


(Jason May) #5

I am thrilled that the Discourse team has been so actively monitoring performance issues in Ember (and Rails, and all the other essential elements of the stack). This is a major chore. Ongoing thanks to @eviltrout, @sam and the whole gang.


(Jeff Atwood) #6

I want to see how bad this is on our worst platform, Android.

Nexus 9

1.7.1

.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed   |  Error  | Samples |  Mean  |
|---------------------------|----------|---------|---------|--------|
| Baseline: Object Create   | 31877.81 |   13.77 |      92 |      0 |
| Baseline: Render List     |   130.06 |    6.32 |     781 |   7.69 |
| Baseline: Handlebars List |   168.12 |    7.89 |     868 |   5.95 |
| Ember.get                 | 18518.96 |    4.65 |      76 |      0 |
| Ember.set                 | 11791.78 |    8.21 |      70 |      0 |
| Ember.Object.create       |   536.58 |     3.7 |      87 |      0 |
| Render List               |     8.29 |  286.22 |      92 |  120.6 |
| Render List (Unbound)     |     15.3 |  178.59 |     150 |  65.35 |
| Render Complex List       |     2.01 | 2725.78 |      26 | 497.42 |
'-------------------------------------------------------------------'

1.8.1

.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed   |  Error  | Samples |  Mean  |
|---------------------------|----------|---------|---------|--------|
| Baseline: Object Create   | 29878.09 |    9.69 |      96 |      0 |
| Baseline: Render List     |   161.48 |       3 |     696 |   6.19 |
| Baseline: Handlebars List |   169.92 |    5.79 |     879 |   5.89 |
| Ember.get                 | 20260.19 |    4.23 |      84 |      0 |
| Ember.set                 | 15649.23 |    2.43 |      84 |      0 |
| Ember.Object.create       |   531.61 |    9.66 |      87 |      0 |
| Render List               |     6.41 |  369.59 |      73 | 155.89 |
| Render List (Unbound)     |    12.91 |   157.3 |     133 |  77.47 |
| Render Complex List       |     1.13 | 4902.25 |      15 |  886.6 |
'-------------------------------------------------------------------'

1.9 beta 4

.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed   |  Error  | Samples |  Mean  |
|---------------------------|----------|---------|---------|--------|
| Baseline: Object Create   |    29074 |   14.58 |      89 |      0 |
| Baseline: Render List     |   136.29 |    3.96 |     809 |   7.34 |
| Baseline: Handlebars List |   159.81 |    7.05 |     863 |   6.26 |
| Ember.get                 | 22785.71 |    5.27 |      83 |      0 |
| Ember.set                 | 16540.74 |    9.77 |      80 |      0 |
| Ember.Object.create       |   512.29 |    7.42 |      69 |      0 |
| Render List               |     6.09 |  504.63 |      70 | 164.31 |
| Render List (Unbound)     |     9.93 |  231.68 |      99 | 100.75 |
| Render Complex List       |     1.21 | 3752.81 |      16 | 826.88 |
'-------------------------------------------------------------------'

1.10 canary

.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed   |  Error  | Samples |  Mean  |
|---------------------------|----------|---------|---------|--------|
| Baseline: Object Create   | 28376.57 |    6.65 |      99 |      0 |
| Baseline: Render List     |    151.2 |    2.99 |     202 |   6.61 |
| Baseline: Handlebars List |   163.94 |    7.49 |     873 |    6.1 |
| Ember.get                 | 23241.43 |     6.5 |      77 |      0 |
| Ember.set                 | 15777.74 |    5.77 |      81 |      0 |
| Ember.Object.create       |   667.26 |    2.94 |      91 |      0 |
| Render List               |     6.77 |  485.85 |      77 | 147.65 |
| Render List (Unbound)     |    11.41 |  222.16 |     116 |  87.67 |
| Render Complex List       |     1.53 | 2455.46 |      20 | 653.05 |
'-------------------------------------------------------------------'

Nexus 7 (2013)

1.7.1


.------------------------------------------------------------------.
|                Ember Performance Suite - Results                 |
|------------------------------------------------------------------|
|           Name            |  Speed  |  Error  | Samples |  Mean  |
|---------------------------|---------|---------|---------|--------|
| Baseline: Object Create   |    7124 |    4.95 |      95 |      0 |
| Baseline: Render List     |   38.73 |     3.4 |     390 |  25.82 |
| Baseline: Handlebars List |   43.54 |   13.56 |     431 |  22.97 |
| Ember.get                 | 3284.22 |    5.36 |      89 |      0 |
| Ember.set                 | 2792.52 |    5.09 |      87 |      0 |
| Ember.Object.create       |   90.75 |    3.33 |      78 |   0.01 |
| Render List               |    1.94 | 2444.73 |      23 | 515.26 |
| Render List (Unbound)     |    4.06 |  562.47 |      42 | 246.07 |
| Render Complex List       |    0.64 | 13497.1 |       9 |   1560 |
'------------------------------------------------------------------'

1.8.1

.------------------------------------------------------------------.
|                Ember Performance Suite - Results                 |
|------------------------------------------------------------------|
|           Name            |  Speed  |  Error  | Samples |  Mean  |
|---------------------------|---------|---------|---------|--------|
| Baseline: Object Create   | 6776.06 |    1.37 |      92 |      0 |
| Baseline: Render List     |    38.3 |    2.99 |     223 |  26.11 |
| Baseline: Handlebars List |    43.3 |    8.98 |     429 |  23.09 |
| Ember.get                 | 3642.67 |   12.03 |      89 |      0 |
| Ember.set                 | 3125.36 |    1.78 |      89 |      0 |
| Ember.Object.create       |    89.7 |    3.68 |      77 |   0.01 |
| Render List               |    2.35 | 1790.95 |      28 |  425.5 |
| Render List (Unbound)     |    3.92 | 1092.86 |      42 | 255.38 |
| Render Complex List       |    0.42 |   13037 |       6 | 2379.5 |
'------------------------------------------------------------------'

1.9 beta 4

.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed  |  Error  | Samples |  Mean   |
|---------------------------|---------|---------|---------|---------|
| Baseline: Object Create   | 6714.98 |    1.41 |      86 |       0 |
| Baseline: Render List     |   36.85 |    5.54 |     380 |   27.14 |
| Baseline: Handlebars List |   43.66 |    8.69 |     429 |    22.9 |
| Ember.get                 | 3475.68 |    6.48 |      86 |       0 |
| Ember.set                 | 3202.18 |    3.31 |      88 |       0 |
| Ember.Object.create       |  109.16 |    3.71 |      83 |    0.01 |
| Render List               |    1.96 | 1702.98 |      23 |  509.91 |
| Render List (Unbound)     |    2.95 | 1121.75 |      30 |   338.6 |
| Render Complex List       |    0.52 | 7909.04 |       7 | 1914.29 |
'-------------------------------------------------------------------'

1.10 canary


.-------------------------------------------------------------------.
|                 Ember Performance Suite - Results                 |
|-------------------------------------------------------------------|
|           Name            |  Speed  |  Error  | Samples |  Mean   |
|---------------------------|---------|---------|---------|---------|
| Baseline: Object Create   | 6520.24 |    5.94 |      97 |       0 |
| Baseline: Render List     |   38.41 |    3.56 |     392 |   26.03 |
| Baseline: Handlebars List |   43.15 |     8.3 |     425 |   23.17 |
| Ember.get                 | 3335.36 |   12.84 |      89 |       0 |
| Ember.set                 |  3030.5 |    3.86 |      86 |       0 |
| Ember.Object.create       |  111.83 |    7.12 |      78 |    0.01 |
| Render List               |    2.36 | 1646.31 |      27 |  423.26 |
| Render List (Unbound)     |    3.47 |  725.34 |      36 |  288.33 |
| Render Complex List       |    0.59 | 5971.48 |       8 | 1699.38 |
'-------------------------------------------------------------------'

(Jeff Atwood) #7

These render complex list scores are damn scary @eviltrout when going from 1.7 to 1.8:

  • Nexus 9 — 447ms → 887ms
  • Nexus 7 — 1560ms → 2380ms

To the extent that Discourse perf resembles complex list (and all previous evidence says it does) we are about 2x slower now on Android since the switch to Ember 1.8.

Android was already 3x to 5x slower than ios, this means we are now 6x to 10x slower.


(Sam Saffron) #8

I applied a few optimisations today

https://github.com/discourse/discourse/commit/0542c080b3edaa95d8c43a23240aa3ee3c70ca23

Here are the results of this optimisation on my Nexus 7

Before

6100ms ~ 5500ms

(Desktop Chrome 500ms ~ 600ms)

After

Render time on long topic: 5480 ~ 5200

(Desktop Chrome: 420ms ~ 600ms)


(note: times above only include rendering, you need to add asset download and parse times and a bunch of other prep work)

Rendering a single post on android is taking about 50-100ms which is insane.

Some other crazy things:

  • Rendering user card (not displayed yet): 168ms
  • Rendering share dialog (not displayed yet): 118ms
  • Rendering composer (not displayed yet): 126ms

So we got a bunch of easy wins to cut off some more time

Looking through profiles, it confirms what we have known for ages, doing anything that involves Ember objects in Android is slow, hence the impact of unbinding ifs (avoids allocation of View class) and using raw rendering instead of components (avoids allocation of View class)

I am pretty confident we can shave this down to 4000ms in a few days of work, we also need to see the impact of upgrading to 1.9, which was just released but does not work on Discourse. If complex bench is anything to go by we should expect 2 to 3 second render times for a topic page with 1.9 which will be a fair bit faster than it used to be.

There is plenty to do on our side but the unfortunate rule we need to follow is avoid creating too many ember views and classes which gets very tricky … considering that trivial code like {{#if something}}test{{/if}} will generate an Ember class.


(Robin Ward) #9

We definitely have to be careful with the raw handlebars approach. As mentioned previously I’ve been warned several times that it might make things worse in the long term for us.


(Sam Saffron) #10

I fixed a lot of the remaining issues around upgrading to 1.9, from my local testing it does not seem that much faster on android than 1.8 was for the topic and topic list.

To test I added this little script to the header (in both mobile and desktop):

<script>
    Ember.run.backburner.run = window.probes.measure(Ember.run.backburner.run,"run loop");
    setInterval(function(){
        window.probes.displayProbes();
    },2000);
</script>

you can preview on meta by visiting Discourse Meta (open in new tab)

To clear preview go to Discourse Meta

Lets ensure we gather some proper android numbers prior to next major upgrade.

Some current numbers on my nexus 7 (ember 1.8.1):

Home page incognito (sample times):

3562ms
3643ms
4130ms

(for comparison, iPhone 5s safari 796ms, 951ms)

Why is Discourse slow on android Why is Discourse so slow on Android? (sample times) (first run loop incognito)

6109ms
6307ms
5714ms

Page stabalises after 15 run loops at:

8050ms
8791ms
8251ms

For comparison iPhone 5s ( 1600ms ~ 2000ms - 9 run loops)

Its harder to profile topic pages cause there is a pile of background rendering, but it is quite interesting to see that 15 runloops in we reach 8075ms

Cloaked view can definately do a bit better on android, trouble is that at 50-100ms render times per post, you would be trading very jerky scrolling with a faster initial view

NOTE

This is only counting time in backburner, which is the Ember run loop, it is not counting

  1. Time to make all the HTTP reqs
  2. Time to parse all the JS files
  3. JS time not in run loop (some exists)

(Sam Saffron) #11

@codinghorror it seems that now that chrome has web view on iOS 8 we are getting stuff like this

Chrome

Safari

Can you confirm?


(Blake Erickson) #12

This is on my iPhone 5s

Chrome:

Safari:


(Robin Ward) #13

The problem with measuring the runloop time is it encompasses way more than just rendering. For example, all our AJAX, all promises, all timeouts go through the runloop. You are benchmarking network speeds here too which are even more unpredictable than JS runtimes.

edit: I messed up, network is not, but a lot of extra non-rendering junk still is


(Sam Saffron) #14

FYI I confirmed with Robin and AJAX network times are not being included in my measurements here.


(Jeff Atwood) #15

@eviltrout one thing I want on our list for this beta release (tomorrow) if we detect an Android device, cut the amount of work in half:

  • if loading 20 topics in the topic list, only load 10

  • if loading 20 replies in a topic, only load 10


(Sam Saffron) #16

Just did another round perf / refactoring work on the topic list…

Go from New -> Latest

Before - on my nexus 7:

1197ms, 1331ms, 1195ms, 1100ms

After refactor on my nexus 7:

520ms, 500ms, 522ms


So our topic list itself is twice as fast as it used to be.

A very large amount of time is still in teardown, but its a lot faster than it was before


(Jeff Atwood) #17