More then 3 million enqueued jobs show in sidekiq? How to I stop it generate the queue?

Support
1

now 3k after I just cleaned all of them

The image shows a collection of jobs for a chat summary, with details about user IDs and force respect seen recently in settings. (Captioned by AI)
The image shows a collection of jobs for a chat summary, with details about user IDs and force respect seen recently in settings. (Captioned by AI)2452×342 112 KB

I try to look up any chat or summary settings in the admin dashboard but nothing was found.

Any idea where they come from? :sob:

P.s. those jobs make my site extremely slow :sob:

1 Like
2

What is in Retries tab of sidekiq?

The image shows the state of a task queue with 37,170.868 processed tasks, 27,055 failed tasks, 6 enqueued tasks, 4 scheduled tasks, and 4 dead tasks. (Captioned by AI)
The image shows the state of a task queue with 37,170.868 processed tasks, 27,055 failed tasks, 6 enqueued tasks, 4 scheduled tasks, and 4 dead tasks. (Captioned by AI)1356×120 15.2 KB

Here there are 6 jobs being retried. That’s an ok number and I know why.

If this is in the 1000’s that’s not good and you need to take action.

Have you got any errors in Email Logs or the Server Error Logs /logs?

Things to think about:

Is your email service provider throttling you? Have you exceeded their rate limits for the level of service you are on? Have you paid your bill with them?

3 Likes
3

A summary of job statuses shows 11,222,546 processed tasks, 6 scheduled, and no retries found. (Captioned by AI)
A summary of job statuses shows 11,222,546 processed tasks, 6 scheduled, and no retries found. (Captioned by AI)2486×602 120 KB

it’s zero

A screenshot displaying a log file with various system messages and error notifications. (Captioned by AI)
A screenshot displaying a log file with various system messages and error notifications. (Captioned by AI)1920×1064 224 KB

and doesn’t seem to have email-related issues in the log.

Is there any setting related to chat summary I can turn off or I should just turn of the chat ?

4

Is the entire queue all UserEmail jobs (for the most part)?

Check your email service dashboard (over at your email service provider, not on Discourse) and see if there is any information there

1 Like
5

yes, almost all of them are user email job

I checked mailgun (4 running instances under my account)

the first one is the only one that runs into this issue and the date doesn’t seem to be bad, I checked logs too but found nothing suspicious either.

A table displaying various metrics for different mail domains, such as percentage of accepted, delivered, bounced, opened, clicked, and complained emails. (Captioned by AI)
A table displaying various metrics for different mail domains, such as percentage of accepted, delivered, bounced, opened, clicked, and complained emails. (Captioned by AI)2536×960 117 KB

6

Do they have an audit trail, can you see a tick-over of continuously sent mails?

Last email sent?

7

A table displaying email delivery status, with all emails marked as "Delivered" or "Accepted" with the message "OK" or "(...)". (Captioned by AI)
A table displaying email delivery status, with all emails marked as "Delivered" or "Accepted" with the message "OK" or "(...)". (Captioned by AI)2518×1396 175 KB

like this?

and the last one should be 12 mins ago

8

So some are getting through, but is it processing them fast enough?

9

how do i check that?

on discourse (admin/email/sent) I saw the chat summary email was send with a low frequency like at most 5-6 per day, most time only 1 or none. :sob: how come it create so many jobs in sidekiq?

1 Like
10

Take a look at your email logs in Admin 
 that might give you more clarity

11

One thing I noticed is that the sidekiq queue issue started right after I upgraded to 3.4.0.beta4-dev, the other instance I installed around the same time seems to have the issue too (confirming now)

also looked at the ids, it seems the system is trying to send chat summary to every users :thinking:
is it normal?

A table displaying the current jobs in low priority, including their arguments. (Captioned by AI)
A table displaying the current jobs in low priority, including their arguments. (Captioned by AI)1060×1128 91.2 KB

update:
Turning off chat prevents those jobs from being queued again, but my site kind of needs the chat function, and I am still looking into other solutions.

another update:
I turned off the chat function for 2 days, turned it on again, and changed the Chat channel retention days from 180 to 3 days. Now it seems to work fine, will keep an eye on it still.

update 02/01:

I set the retention days to 7 , aound 1,101,685 Enqueued jobs right now, seems my server is able to handle this amount and the site won’t become slow.

4 Likes
12

@th21 how many users and chat channels do you have?

How “fast” (or slowly :cry:) does the low queue get processed?

13

9k users and 3 channels (including 1 default staff channel)

this is the current status

The image shows a dashboard with 40,301,858 processed items, 40 failed attempts, 5 busy instances, and 5,522 enqueued items across four queues labeled critical, default, low, and ultra_low, sorted by latency and size. (Captioned by AI)
The image shows a dashboard with 40,301,858 processed items, 40 failed attempts, 5 busy instances, and 5,522 enqueued items across four queues labeled critical, default, low, and ultra_low, sorted by latency and size. (Captioned by AI)1430×860 105 KB

before I made the changes mentioned above, the low queue size seems never decrease

14

Sorry about that @th21 - it was indeed a bug that will be fixed by

We have a query that list all the “users with unread” mentions or dms for chat, and, in trying to be too clever, it ended being wrong. Instead of enqueue a “chat summary” (aka. email notification) for mentions in channels or unread in dms, it was enqueueing a job for every messages :expressionless: Since that job ran every 5 minutes, it easily bloated sidekiq with useless jobs
 (note: the email were thankfully not being sent as we would not be able to retrieve any mentions).

3 Likes
15

thx, When is the estimated time of arrival for this commit to be merged into test-passed?

1 Like
16

< 48 hours

1 Like
17

@th21 it’s been reviewed and merged :+1: