Note: I am not certain if this is a bug in Discourse. I have tried to collect necessary evidence, and so far I have not found anything pointing to our infra/setup. Our configuration is as vanilla as as possible at Tappara.co Observed phenomenom: Chat-like rapid discussion topics stop updating automatically. After a delay of 30 - 180 secs the updating usually resumes, revealing the posts that were made during the freeze. What we know so far We did not see this during the previous season, the last game was played in March. We run stable branch and did the latest major update in August. The issue was immediately reported in the first exhibition games, with moderate traffic/activity. This impacts iOS and Android Chrome, but is far less frequent on Chromebook. As I write this, I am seeing freezes on my Android phone, while the discussion flows as expected on my Chromebook. Two different devices in the same network. The experience varies per user/client. Different users report the freezes at different times. Overall we just recorded roughly 300 messages in about 30 minutes and users reported dozens of freezes. Mostly the freezes seem to correlate with events in the game (goals, penalties). Things I have tried to rule out CloudFlare – we did one game without CF caching, and the issue persisted. CPU overload – CPU usage is well within limits, usually hovering around 20-30%. Disk exhaustion – Disk I/O seems to be well within limits. We have UpCloud’s MaxIOPS SSD’s. Other info I had the Chrome inspector running during the game and some 429’s were recorded, but for me they did not correlate with freezing. The end users are not receiving the notifications regarding 429’s (slow down) or extreme load. The updating just freezes, and then just resumes. Has the rate limiter changed recently - I am under the impression that ratelimits should trigger a notice on the UI? A really nasty problem, that really hurts the play-by-play game chats. We have been running these for years, and I have never seen this before.

[imagem] Falco: solution: :moneybag: Solution: Make leaner and meaner code :wink: So if Redis is the bottleneck, how would you scale horizontally? It still puzzles me what has changed since last season. I can’t see that much organic growth, or increase in game chat popularity. Still our capability to serve has reduced dramatically, and is choking even in the calmest games.

[imagem] ljpp: It still puzzles me what has changed since last season. Until you can collect metrics on your historic instance of Discourse and then compare to the metrics you collect on your current install, while maintaining the exact same hardware, this will remain a mystery The whole difference could be that your VPS provider shifted you from one physical machine to another, or that you acquired a noisy neighbour, or that your VPS is now running 17 vs 13 avg number of co hosted services per machine

Please do not speculate on pushing the issue to the VPS provider. UpCloud is one of the best on the market, and they have checked their end for anything out of the ordinary. They advertise on our site and it is not very good PR to have the site stuttering :smiley: But there is no historical data, and TBH I was not paying that much attention as everything just worked, until the first exhibition games took place in August. Of course the behavioral patterns of humans have changed thanks to COVID, and who knows what else. I can’t see it in the metrics of our site or server, though. :man_shrugging: But this is excellent testing material. Just provided @riking some screenshots on what happens when the server overloading kicks in. I guess you guys don’t see it that often.

Note that nobody is disagreeing with you – we’re just pointing out that a doctor can only do so much to diagnose a patient when the doctor is limited to seeing the patient through a video camera on the internet.. :movie_camera:

[imagem] ljpp: A notification that he is logged out He is brought to the site index page The index page has the banner notification of high load Just wanted to say this was exactly as I experienced when I first set up my site (so its not unique to your site). Heres a thread I made about it at the time: [imagem] Users kicked out of topic, more memory required? Support Today my forum had its first bike spike in traffic, and users experienced their first big issue. They were receiving logged out messages every 5-10 minutes, returned to the home screen, but in most instances remained logged in. My forum is DO basic droplet, shared CPU, 1GB ram and has been live for about a week. The spike occurred during the match were 400 posts are made in a single thread over 90 minutes (as people followed the game). It appeared there were 40-60 posters online during this… This is what caused me to jump up different CPU/Memory options outlined here [imagem] Real-time updating of topics freezes under high activity Support Mine is a football forum and I have experienced similar challenges. Basically what I found was it was scalable issue. The issues for me kicked in at different levels. Digital Ocean 1 CPU 1 GB = 30 -40 users in chat like situation 2 CPUs and 2 GB= 70-80 users in chat like situation 4 CPUs and 8 GB = fine for 120 users and 1000 posts in 2 hours. Didn’t reach limit. I am trying the different step up levels with Hetzner (Mirroring site) as cheaper and didn’t go as smoothly as hoped. My expe… Unfortunately, I have not had a chance to properly swap to Hetzner from Digital ocean as I described (started a new job). But will do as soon as I get a chance this month. The end-user experience of being kicked out of the thread, or remaining in the thread (with the logged our message). Did seem to correlation dependent on load. (more users were sent to site index after a goal scored) I don’t have enough technical knowledge to be helpful, but felt it might help to know a sporting site with similar peaks of chat like behaviour does lead to a similar issue. But mine (smaller, and younger site) was resolved by further upgrading server.

[imagem] ljpp: But there is no historical data, If you’re interested in having data to make decisions about how to diagnose things going forward, you can install Prometheus exporter plugin for Discourse.

[imagem] Falco: Well, at least that has a know solution: :moneybag: Just a brief update: Installed a new 2 container environment on 2 VPS servers (web_only, data). Surprisingly (for me) the web_only server is exhausting, while the data is relatively lightly loaded. Both running a 4x vCore 8GB RAM UpCloud.com plan. Upgraded the web_only to a 6x vCore / 16GB RAM UpCloud.com plan. Increased Unicorns to 18. Still we are hitting various 429 limiters. The system under high load -mode did not kick in though.

The hockey season is ruined by the COVID, and they are now playing a few random games without audience. Since we do have hosting credits with UpCloud.com, we are pushing to improve the experience using what we got. Now running the 6x vCore 16GB for web_only and 4x vCore 8GB for data, unicorns at 18. We once again disabled the ratelimiter… DISCOURSE_MAX_REQS_PER_IP_MODE : none …which helps, but we still get 429’s from POLLs, that produce the long delay/freeze for the end user. We are going to continue tweaking by increasing the DISCOURSE_REJECT_MESSAGE_BUS_QUEUE_SECONDS. But before we do that, a question to @sam / staff: Is there an environtment variable to increase the thresold for extreme load - read only mode -limiter, or can it be disabled completely?

[imagem] ljpp: Is there an environtment variable to increase the thresold for extreme load - read only mode -limiter, or can it be disabled completely? This should not be needed, we would love to host you so we can get to the bottom of why this keeps tripping you even though you have such low traffic.

[imagem] sam: This should not be needed Perhaps so, but we would like to be slightly less protective over the server as the naturally occuring activity spikes are very short, and generally stabilize within a minute or so. So adjusting the thresolds just a little bit higher might improve the UX, while waiting for the move.

Atualização em tempo real de tópicos trava sob alta atividade

Suporte ao Produto

pfaffman (Jay Pfaffman) 2 Dezembro , 2020 13:01 59

Se você estiver interessado em ter dados para tomar decisões sobre como diagnosticar coisas no futuro, você pode instalar o plugin exportador Prometheus para Discourse.

Tópico		Respostas	Vistas
Improving Instance Performance (Megatopics, Database Size and Extreme Load) Self-hosting	60	6755	13 Outubro , 2020
Extreme load error Self-hosting	19	2059	13 Agosto , 2023
Topic history not loading correctly after migration Product Support	20	1279	8 Fevereiro , 2023
Discourse installation has been getting slower and slower and slower Self-hosting server-resources	36	2287	15 Abril , 2023
The MEGATOPIC: public good, or public menace? Community Building	41	16968	4 Abril , 2021

Atualização em tempo real de tópicos trava sob alta atividade

Related topics