After yesterday's upgrade, my site is taking 50 seconds to open (and then behaves normally)

[Bathinda (Bathinda Helper)]

Before updating my website to v3.4.0 Beta3-dev- (5e86bc2f43) all was ok.

But since this update yesterday, my site is taking nearly a minute, or more, just to open.

• As a diagnose step, I diabled all theme components and ALL plugins and REBUILT MY Web_Only container, and yet same problem is there.
• But if I run in safe mode, checking all 3 boxes then also only it loads faster than before, but still taking nearly 30 seconds. Mind that if I check only first box and deselect other two, or vice versa, then no benefit is gained in speed.

Pls help.

[Bathinda (Bathinda Helper)]

You won’t believe, because even I can’t believe that just 5 minutes after this post creation, (and just switching my own/admin’s interface theme to some other, and then reverting back), solved the problem.

I.e. wo any updating, or rebuilding, nothing, the site has started opening ok (full 24 hours passed during the problem). I don’t know if I should delete this topic or not (I’m anyway closing this).

Edit: The problem resurfaced next day. Sometimes there, sometimes not. Would report futher down.

[Ed_S]

(It’s possible that some migration task was running for a while after the update.)

[Bathinda (Bathinda Helper)]

Thanks @Ed_S
But today again, for 5 minutes I found the same problem. When this problem occured today, I tried opening my site as another user (who is ‘Moderator’) in the incognito chrome mode. There also problem was present but only 50% (it was opening in half time, around 20-30 seconds).

And then I opened on my mobile, same problem was there also. But after half an hour, when I opened my site, all was working ok.

[Ed_S]

Perhaps try

@Ed_S

my list of standard instant diagnostics:

cat /etc/lsb-release
uptime
df -h /
free
vmstat 5 5
dmesg|egrep -i "memory|oom|kill"
ps auxrc

But run these (at the command line on the machine which runs the server) at the same time as browsing the site to provoke the slowdown.

[Ed_S]

Also worthwhile to enable and use the mini-profiler, and report your findings.

See

@tobiaseigen

The below seems to be a good step by step guide

[Bathinda (Bathinda Helper)]

@Ed_S

dmesg|egrep -i "memory|oom|kill"

Thank you very much.
If you’ve got a moment, pls see below. Though to me, all findings are very much correct:

[Bathinda (Bathinda Helper)]

@Ed_S

Also worthwhile to enable and use the mini-profiler, and report your findings.

There were a things in the ‘Mini Profiler’ topic which I didn’t understand fully. So, I’d go for that solution, if I MUST.

[Ed_S]

Thanks for running the diagnostics. The main thing I see there is a huge number of writes to disk. But I can’t guess as to the cause.

[Ed_S]

Maybe it’s worth installing and running iotop. We might be able to see which process is writing a lot of data to disk.

apt install iotop-c
iotop -o -b -n 22

[Bathinda (Bathinda Helper)]

Thanks.
For info, my ubuntu server only has this sole website installed on it. And that too with very few users, only 3-4 users visit the site daily. So whatever activity the server might have, should be back ground activity only.

2nd, I ran the iotop command just after rebuilding the web_only container, if that be relevant in some case. And also, website is opening almost 97% fast (is opening in 5 seconds for myself/admin). When problem surfaces, it starts taking more than 30 seconds upon hard-refresh (Ctrl+F5).

Just one line of the results might be missing from between the 1st screen shot and the 2nd.

Thanks for helping me.

[tobiaseigen]

Did you install discourse using the official install instructions?

[Ed_S]

I don’t think I see anything too odd in those iotop outputs.

Perhaps worthwhile to explore the sidekiq situation: various tabs on your forum.url/sidekiq page (which can only be seen by an admin account)

[Bathinda (Bathinda Helper)]

@tobiaseigen

Did you install discourse using the official install instructions?

Yes. And my site was running ok for last 4 years. Only after updating it the before day I opened this topic I noticed that it is behaving like this (when hard refreshing with Ctrl+F5 as an admin, taking 5 to 50 seconds in the diff times of the day, afterwards works normally).

Today again too, I’ve checked logging in as diff users in the latest versions of chrome, on my mobile also (but under the same wifi), and found 5 to 30 or 50 seconds to open.

[Bathinda (Bathinda Helper)]

Main parent domain name showed this sidekiq (zoomed out a bit to accomdate whole of it):

zoomed in a bit:

[Ed_S]

That looks like a huge number of failed jobs, which might be significant. Perhaps someone else has the expertise to help diagnose.

Here’s mine for comparison - quite a low-traffic forum.

[Bathinda (Bathinda Helper)]

@Ed_S

That looks like a huge number of failed jobs, which might be significant. Perhaps someone else has the expertise to help diagnose.

Could someone pls help me about so many ‘Failure Jobs’ in side kiq?

[Ed_S]

please could you share screenshots of all seven sidekiq status tabs? On the sidekiq page, at the top, you should see
Dashboard Busy Queues Retries Scheduled Dead Scheduler

[Bathinda (Bathinda Helper)]

• Dashboard:
  

  

• Busy:
  

  

• Queues
  

  

• Retries:
  

  
  
  
  
  
  
  

And as many more skipped.

• Scheduled:
  

  

• Dead
  

  

• Scheduler
  

  

Above screen shot didn’t accomodate all.

[Ed_S]

Thanks - I’m no expert but I see two kinds of jobs featuring heavily: PostSentimentAnalysis and GenerateEmbeddings.

I wonder if it’s relevant that you have a mix of English and Hindi in your content. I’d expect that to be fully supported, but also it might be relatively uncommon.