Sidekiq queue keeps growing

support
1

Hi,

I recently set up a discourse, and migrated content from a phpbb3 forum.
We actually migrated jost some topics/subfora, which means that after a full import, we deleted lots of categories again, to only keep the subfora that got split out. Deleting those postings and categories happened immediately after the import.

Now, after that import, sideqik showed around 100k jobs that needed to be processed. After a couple of days we are now at 500k processed jobs and around 165k still to process.

Screenshot:

So they get processed, but it seems every X hours, another batch of around 100k jobs get created. It does not seem like this will get purged eventually.

Is this normal behaviour? Can the deletion of messages be part of a problem here?

Thanks,

2

What types of jobs are in the queue?

1 Like
3

currently all in the default queue, they all have the form

Jobs::UserEmail	[{"type"=>"digest", "user_id"=>9192, "current_site_id"=>"default"}]
4

Nuclear solution would be to flush redis. You’d lose some emails but you would be in a much better state.

5

Better than having some providers (or your own) blacklist you. We had this happen when we went live. The first couple hours or so were a bit hectic until we figured out what was going on.

Basically we were sending a digest to every single user. All 39K of them. Given that they all had to reactivate their accounts by sending a forgot password mail while this was going on there was a certain level of… confusion.

1 Like
6

Ouch, better flush that asap then. Can I just delete the queues via the sidekiq web interface, or is there some manual way that I should do that?

7

I can’t remember exactly what we did. The first step was to pause all outgoing on our mailserver. Then I’m not sure what we did next.

@zogstrip will probably have a more complete, not to mention elegant, solution.

Check your bounced messages first. As long as you’re not generating enough bounces at any individual domain you may be under the radar.

We managed to get verizon to blacklist us iirc.

Not. Good.

1 Like
8

ssh into your server and then

cd /var/discourse
./launcher enter app
redis-cli flushall
4 Likes
9

Done. Thanks you.

Is there a better way to handle this, or to avoid this? We plan another phpbb migration in the near future…

10

I would flush every time you do a migration.

5 Likes
11

It sames now, at every job run Jobs::EnqueueDigestEmails about 1500 new digest mails are loaded into the queue.

I’m not sure if I should keep flushing this queue now. I also do not understand why, since the migration, there have been 100kish messages, given the forum had around 10k users.

What exactly is happening here?

12

Something’s definitely wrong here… Switching to PM.

13

Well, for me it was the same but with SMF and with over 750k enqueued jobs and growing even though I set it up to 50 threads… I guess the nuclear option worked.

14

Has this been solved since then? Any insights?

15

What jobs are queued? The nuclear option is just to nuke sidekiq and start from scratch.

./launcher enter app
redis-cli
flushall
4 Likes
17

This should be mostly solved now, with imports disabling emails for imported users.

5 Likes