Real-time updating of topics freezes under high activity

The 1st game weekend went alright from technical point of view (less so if we look at how the team played).

One phenomena is still reproducable. If you multitask with your phone, switch to other apps during the game chat and then come back, you sometimes end up in a situation where the topic is not updated. Status avatars do blink, indicating there is a a constant flow of posts.

A browser refresh or a visit via the index page is needed to see the missed posts.

All this using an up to date Android device w. Chrome.

2 Likes

Thanks for helping us figure these incidents out – it’s a process.

2 Likes

We had a derby game last evening, so it was an great opportunity to monitor this:

  • One senior user logged the whole game with Chrome Dev Tools network inspector, and no error statuses were reported on the client :+1:
  • I ran a quick poll during the game, got a few dozen responses which were 94% postivive (no issues). This is a very good result and I am not sure if a clean 100% is even possible with this survey method (random users, random HW, random network conditions). So I consider anything above 90% a success. :+1:

To sum it up, on the CDCK SaaS infra this is now a marginal issue and the features generally work well even under a heavy/spiky load. Give me a nod if you wish to deep dive into this, and I’ll provide a potential date and time. After all you never know what happens in a game, and how the audience reacts to the events.

COVID-restrictions have eased up, so currently the arenas can be sold to full capacity. This has an impact on our audience as the HC fans will be on-site at the games, rather that furiously chatting online.

2 Likes

We merged a performance patch that will affect sites that are seeing lots of activity, specifically when many users are in a single topic and posts receive likes. During those events one of the possible outcomes was this exact “freezing” behavior.

In a single site we host, this change resulted in 12M less requests for the posts endpoint being made per event day, which competed with the live updates of newer replies.

4 Likes

Noted. We have a back to back derby on Fri-Sat, so there will be a buzz. Although 13.000 people will be at the grand opening of our new arena, which may have an impact on our load, positive or negative.

Will monitor, short & long term :+1:

4 Likes

If someone at CDCK has interest in deep logging, we are expecting a buzz tonight at 18:30 - 21:00 and especially tomorrow at 17:00 - 19:30. Finland local time UTC+2.

1 Like

Everything going OK these days?

Let me get back to you in 3-4 hours.

Generally this has not been an issue (success rate >90%) or a source of complaints recently. On the other hand our regular users know what to do if it happens (a quick refresh or visit to the index page), so they might not be paying attention.

Anyway, we have a game starting in 45 minutes (17:00 UTC+2) so I’ll run an user survey on the matter and possibly some client side logging as well.

@codinghorror Here are the results.

We ran a poll, like we have done many times before. This time the questions were written in the London language, so you folks can read it too. I asked users for full stop freezes and the type of client device they were using. Findings:

  • Low number of responses, high margin of error
  • Roughly 85/15 result. This is exactly in line with previous polls. No changes for the better or worse
  • I was able to catch up with couple of the users reporting freezes. Indeed they did experience a full-stop type of freeze, which can be worked around by refreshing the browser or visiting the index page. So it is still possible to reproduce this.

The game took place at an earlier time than usual and was actually moved to this date due to the COVID-mess. Therefore the number of concurrent users was relatively low. Under normal conditions I would have expected a higher number of users and posts, as the game was an away game, 7 goals were scored and the end result was a bitter loss.

2 Likes

It’s been a while, but accidently reproed this yesterday. An intense game and a dramatic loss 6 seconds before the buzzer caused heavy chat spikes, and my view stopped updating. I was not able recover by visiting the index page and had to refresh the browser.

So it is still there, although not very common. As discussed elsewhere, we are ready to pilot the chat as soon as it has a some sort of topic archiving feature available.

1 Like