Forum is fast, notifications are delayed by SMTP delivery, what does it mean?


(Anton) #1

So, the season of max activity in our community has begun, as winter has come. This is usual for forum.kozovod.com

The forum is pretty fast and works instantly.

But for some reason, notifications (blue circle) are often very delayed. I’m talking about a 3-10 minutes delay.

If the server can’t bear with the load, I expect the forum itself be slow as well; but the fact is only notifications are “slow” (= delayed). What can it mean?

Can’t see any particular reasons for such a behaviour in Digital Ocean monitoring:


(Michael Howell) #2

I blame Sidekiq. Open https://forum.kozovod.com/sidekiq/


(Jeff Atwood) #3

Yeah what @notriddle said; notifications are all about the sidekiq. Check your sidekiq queues and see what is gunking up the works.


(Anton) #4

Looking at recurring jobs history, I can notice a few jobs that take long to execute. Is this time usual for these jobs, or is it too long?

Jobs::DashboardStats
7 minutes ago
OK
23.43 secs

Jobs::PeriodicalUpdates
9 minutes ago
OK
7.24 secs

Jobs::Tl3Promotions
21 hours ago
OK
36.89 secs

Jobs::CategoryStats
13 hours ago
OK
37.64 secs

Jobs::TopRefreshOlder
6 hours ago
OK
55.22 secs
(this one should be the concert since it’s run once per 24 hours only)

There’s also one dead job, 9 months old, is it safe to delete it?

5630 enqueued, is it normal?


(Anton) #5

Just an idea, can some jobs take priority over others?

For example, notification should have a higher priority than digest. For instance, weights can be used, e.g. for every 5 notification jobs only one digest job is executed.


(Jeff Atwood) #6

This is already how it works.


(Anton) #7

So, why all jobs related to emails can be executing so slow. Might it be my email provider which operates slowly, or Discourse building digest email takes bigger part of the process and could be sped up by increasing the processing power of VPS?

For instance, I can imagine building a digest email is more than one SQL query, all run individually for every single email. In this case, it can be speed up just be tuning VPS or buying a better VPS.

Re

Quick question, can the weights be tuned? E.g. in app.yml? For instance, I consider digests are good for our community but nowhere near being a priority. People are reporting delayed notifications, so I’d give notifications much higher weight. P.S. As a work-around, I temporary switched off digests altogether in the Admin panel.


(Jeff Atwood) #8

Maybe your mail service (the connection, negotiation, etc) is slow? I dunno, maybe @sam or @tgxworld can advise.


(Sam Saffron) #9

You have 5,630 jobs queued, so yeah that would make everything slow. That backlog needs to clear up for stuff to go back to normal.


(Alan Tan) #10

@meglio Under

Are there any jobs that are stucked in the queue? If there isn’t, you can try bumping UNICORN_SIDEKIQS to 2 and see if it helps.


(Anton) #11

The Busy section showed that the jobs were processed quickly enough, nothing was stuck for a long time. However, it took ~10 seconds for a lot of email jobs. For instance, with 5sec polling interval, I saw many email related jobs saying that the execution had started 7sec, 8sec or even more ago.

After turning digests OFF AND ./launcher rebuild app, it eventually processed the queue and went down to 0 queued items. So it now looks unrealistic that just digest emails stuck the whole queue and let it grow that much. PM emails are still ON, and I can’t see and jobs taking longer than a few seconds — the Busy tab shows zero items even with polling set to 2 seconds.

This is now confusing. CPU never went over 70% during the growth of that 5k items queue, so it looks unrealistic that building digests individually per-user to prepare emails couldn’t be processed in time. Or… it’s just my lack of understanding of how hardware, or Ruby, or Discourse, works.


(Rafael dos Santos Silva) #12

It sounds like your SMTP service was having a bad day, because you shouldn’t take more than 1s to send the email.

If this happens again, try switching SMTP providers.


(Anton) #13

Today the same issue came in but only after enabling digests. There were no issues with sending notification emails, but digests take more than 10 seconds:

Trying to understand how can it be that only digest emails take that much time, but not the others.

If it’s not a digest generation speed issue, can it be that the SMTP provider checks whether the email is spammy with some 3rd party service, and it takes that much time because digest emails are very similar to each other and thus may look suspicious to the SMTP provider? Any other ideas?