Optimizing Performance for Non Logged In Users on Mobile


(James Mc Mahon) #1

I’ll preface this by saying I do understand that Discourse is a web application rather than merely a website.

But, I’d really like to get the load times down for our mobile users and I’m looking for ideas on how we can make that happen. Currently the site is “behind” (Free) Cloudflare so we’re using SSL+HTTP2 between the edge server and the customer.

TTFB is a bit long at 1.3s and I’m open to hearing some things to check on the server for that. I can look in Nginx logs too, but I’ll need help on where to look.
Total load time is 9seconds and it’s the objects that follow that really hold up the show.

Some examples on a 3G connection:

Landing page: WebPagetest Test Details - Dulles : forums.kartpulse.com - 09/08/17 19:38:37

Typical Topic w/o images: WebPagetest Test Details - Dulles : forums....back-for-2017/1135 - 09/08/17 19:44:08

I’ve considered offloading images to S3/cloudfront, but then again in these two examples that woudn’t have an impact.

Server is provided by Interserver. It’s a VPS with 4GB of ram. Not sure how many cores are available to me, so I’ll have to check that. [edit, running cat /proc/cpuinfo | awk ‘/^processor/{print $3}’ gave me a 0, so looks like a single core.]. Nonetheless, it seems rarely taxed.

Grateful for backend and frontend ideas to speed this up. Of course I’ll share anything I uncover here as well.

Here’s my plugin list:

discourse-details 0.4 N Settings
guest-gate 0.2.0 Y
discourse-narrative-bot 0.0.1 Y Settings
discourse-nginx-performance-report 0.1 Y
docker_manager 0.1 Y
lazyYT 1.0.1 Y
poll 0.9 Y


(Jeff Atwood) #2

Switch to some other software other than Discourse, is your only option here.

Discourse is a web application and its JavaScript bundle has to be downloaded and processed before the webapp can start. Nature of the beast. Repeat loads are blindingly fast, since the app is already installed in your browser (cache) at that point, but that first load of “install the app on my phone / tablet” is never free.


(Joseph Coates) #3

If your discourse forum isn’t your main website and most of your first time visitors are coming from your main site, then perhaps you can send them your forum’s JavaScript as soon as your main site loads. By the time they click the link to go to your forum, everything should be set to go.


(James Mc Mahon) #4

Hi Jeff. :slight_smile:

I’d counter that Discourse is the ONLY option, I think it deserves to stay that way which is why I brought this up.

Discourse is being built for the future of discussion on the internet right?

The future of the web (including it’s successful communities) is speed.

  • Users are impatient.
  • Search engines are increasingly emphasizing speed.
  • Facebook is penalizing “slow” sites.

Against this backdrop, our communities are competing for first visits with pages delivered by AMP and FaceBook native articles. So how can Discourse’s communities (directly via it’s own codebase, or indirectly by leveraging other solutions) adapt? There’s a ton of smart people here, I’m sure we can figure out some low hanging fruit.

Putting a community manager hat on for a second.

  • Think about how much work goes on to develop and execute a community strategy.
  • How hard we work to get those folks to come to our community for the first time.
  • Think about how that job of getting folks to click and visit is probably going to get more difficult over time.

Our first visitor clicks are gold

So with that in mind, we need to do everything in our power to make sure they stay around long enough for the page to load, get value from that click and begin their journey with our community.


Thoughts

I guess we could break it down into application performance and “web” performance?
Couple of talking points, thinking about first visits

Reduce TTFB. (That’s application and server performance right there.)

  • DB optimizations or something else code-ish??
  • Perhaps page caching with varnish (or similar) can help reduce TTFB for first time visitors and avoid a DB query in many cases?

Combine JS
I know rocketloader from cloudflare does this, but it seems to need tweaks to which js’ are downloaded so things don’t break. I personally haven’t had luck with it, but perhaps someone else has?

Limit the number of objects downloaded on first visit, or non signed-in users. Send them a “Discourse Lite” if you will. They don’t need the entire gamut of the Discourse application, rather just enough to read the page?

I think Jalopnik (Kinja) has something “similar” to this for their commenting system in Facebook Native Articles. It loads a “lite” version of the article and commenting system, then when you click on a link to comment or interact, it brings you to the full site (Application)

Present the content for the first visit differently via a parser of some sort
Maybe something could be done with AMP to process\present the meat of the content in a cut down manner?


I understand this isn’t trivial work here, I understand that resources are limited, but I do think this could be crucial work for Discourse and a key to it’s continued success.

I’m willing to pitch in on this. I’m a not developer that can code perse, but I am technical and I’m willing to test and contribute.

Let me know what the next steps should be for me to do that.


(James Mc Mahon) #5

If I’m understanding correctly, that would impact the load on the main website, correct?
How much JS are we talking about here?


(Jeff Atwood) #6

Sort of, Google AMP is an unmitigated disaster. Generally Google gives good advice, but sometimes… they don’t.


(Joseph Coates) #7

Not if you loaded it last :thinking:


(Erlend Sogge Heggen) #8

We do plan to do something about this. Some old dev notes:

Our initial load requires a large amount of assets and can be very network heavy, even for anonymous.
(…)
What can we do to speed up first load?

There are a few things we can do to speed up our first load:

  • Send less stuff to clients
  • Perform less JavaScript work evaluating scripts on first load and rendering
  • Start running JavaScript initialization earlier (by sending less stuff and splitting assets)

(Jeff Atwood) #9

Yes, but the text you quoted is along the lines of

we can lose weight by being less heavy

The particulars of how are quite excruciatingly difficult. If you want ultimate speed, just send down static HTML. That’s effectively what Google AMP is!

(Well, except that they require certain mandatory Google JS in it, another reason people hate Google AMP.)


(ljpp) #10

Not an expert in this field, but how is the single core performance of your server? VPSBench gives you a rough idea. My subjective opinion is that single core performance correlates with site responsiveness.


(Jeff Atwood) #11

Everything looks fine in the test you ran:

I compared a webpagetest run with a comparably small topic page on discourse.codinghorror.com which is running on a pretty fast dedicated mini-pc and it’s certainly on par with that, I don’t see anything weird here.

So the thrust of this seems to be

how can we convert Discourse from a JavaScript webapp to something else?

and the answer to that remains…

:wink:


(Jeff Atwood) #12

One thing I did notice that’s odd in the webpagetest results, is that TTFB for my discourse.codinghorror.com instance small topic page was ~1.5 seconds. That seems super high!

I hand checked it in Chrome Incognito F12 using the same URL, and I get…

Notice “stalled” and “proxy negotiation” takes 1 second. Very very suspicious. But notice that Waiting (TTFB) was 173ms @SouperC.

I checked in Firefox Anonymous F12, same URL, and I get…

So “waiting” is TTFB in Firefox, 185ms and that’s consistent with what Chrome says.

Blocked

Time spent in a queue waiting for a network connection.

The browser imposes a limit on the number of simultaneous connections that can be made to a single server. In Firefox this defaults to 6, but can be changed using the network.http.max-persistent-connections-per-server preference. If all connections are in use, the browser can’t download more resources until a connection is released.

@riking what do you make of this? I changed that Firefox setting from 6 to 15, restarted Firefox, confirmed the setting was changed… and it had no real effect on the network trace. Still blocked for ~330ms. I raised it all the way to 50 and virtually the same result.


(Jeff Atwood) #13

Also, as a final point of reference. Your webpagetest result says

Chrome, on a similarly small topic page in Incognito mode, says…

(these numbers seem too low to me, maybe it’s polluted even in incognito mode because I use Chrome as my primary browser and I visit my own site.)

Firefox, on another small topic page in Anonymous mode, says…

These numbers are much, much lower than what webpagetest is reporting.

In case you are curious here is what a repeat load of the same URL looks like in Chrome:

and in Firefox:


(James Mc Mahon) #14

I should clarify that I’m talking about mobile performance over mobile connections. That’s what I ran the speed tests on. I renamed the topic.

70% of cellular network connections globally will occur at 3G or slower speeds through 2020.
~Google

That’s where I was going with AMP, for all it’s faults. Sure, there’s a price to pay I guess, but if it gives a better first time visitor experience…

CPU/Server grunt will impact the TTFB (Time to first byte) mostly. Although not exclusively, dns lookup, redirects, establishing an SSL connection add to the TTFB as well. While the TTFB is long at 1sec+, it looks like it was long for Jeff on his install too. After that though, it’s all down to how long it takes to send the required objects to the user’s browser. Have a look at the waterfall in the links I posted to get a sense.

Good point.

I’m going to re-frame it as “how can we present the results/output of the Discourse js webapp to non logged in users as fast as possible in order to stand the best chance of having them join our communities”


#15

rocket loader ruins the site btw, so not something anyone should enable


(James Mc Mahon) #16

I don’t necessarily agree. The other part of what I said was…

it (cloudflare) seems to need tweaks to which js’ are downloaded so things don’t break

You can customize and suppress object from being combined to prevent issues, but it takes time, testing and effort and in all fairness, the total payload for Discourse is still going to be pretty big after combining .js


(James Mc Mahon) #17

Let me know if I can help with testing or in some other way.
What about optimization for the TTFB, are there any quick wins to be found under the hood in the DB or somewhere else? 1-2sec seems to be the standard across discourse installs.


(Kane York) #18

WPT uses a relatively slow CPU for the JS execution.

In the general case where nothing extremely wrong is going on, block times are pretty much always something browser-internal and you can’t really help them except by just doing less work.

It looks like there is a significant server delay there.

I just tested with the homepage, and also got a long server time:

Clean state (Windows… that proxy time…):
image

New tab in existing session
image

My RTT is extremely low due to physical proximity to the server:

image


(Jeff Atwood) #19

I get similar numbers to Discourse.codinghorror.com which is in Boston, so the time to first byte is around 200 - 300 ms @SouperC and has zip to do with the network.

Feel free to test in your browser yourself using f12 and the network tab if you don’t believe me.

Hard to say what webpagetest is doing, perhaps they are including https overhead in that time to first byte calculation?

  • CPU wise they are mediocre but should not be awful. I can pretty much guarantee than an iPhone 7 will be faster than the server CPU hardware webpagetest is using. Android not so much, but it’ll probably be in the ballpark.

  • network speed is basically a constant, if you have a slow network, then Discourse will be unparalleled on second request (20kb) – but downloading the Discourse JavaScript app bundle on first load is unavoidable.


(Chris Jason) #20

For most of our first-time visitors, the plain html would be fairly enough. If they come through Google, they often just need a specific answer. If they are having a good experience, they are more likely to sign up and join the discussion.

So for me, the perfect solution would be:

  • Use AMP or just static page to compete in SERPs
  • Display a button to load full site

Btw: Reddit does the same - you cannot join the discussion on their AMP site, but often you just need an answer for a specific question. For this purpose the Reddit AMP site is just fine.