The user svg reports that their Sidekiq queue is growing uncontrollably after migrating content from a phpBB3 forum, with hundreds of thousands of jobs still pending. After deleting many categories and posts post-import, the queue continues to accumulate jobs, primarily Jobs::UserEmail for digest emails. zogstrip suggests flushing Redis as a solution, which svg confirms resolves the issue. JagWaugh shares a cautionary tale of sending digest emails to all 39,000 users, leading to email blacklisting. zogstrip recommends flushing Redis after every migration. svg notes that the queue keeps adding ~1,500 digest emails per run, questioning why so many jobs exist despite only ~10,000 users. riking mentions that recent imports now disable emails for imported users, likely addressing this issue. meglio asks if the problem has been resolved.
I recently set up a discourse, and migrated content from a phpbb3 forum.
We actually migrated jost some topics/subfora, which means that after a full import, we deleted lots of categories again, to only keep the subfora that got split out. Deleting those postings and categories happened immediately after the import.
Now, after that import, sideqik showed around 100k jobs that needed to be processed. After a couple of days we are now at 500k processed jobs and around 165k still to process.
Screenshot:
So they get processed, but it seems every X hours, another batch of around 100k jobs get created. It does not seem like this will get purged eventually.
Is this normal behaviour? Can the deletion of messages be part of a problem here?
Better than having some providers (or your own) blacklist you. We had this happen when we went live. The first couple hours or so were a bit hectic until we figured out what was going on.
Basically we were sending a digest to every single user. All 39K of them. Given that they all had to reactivate their accounts by sending a forgot password mail while this was going on there was a certain level of… confusion.
It sames now, at every job run Jobs::EnqueueDigestEmails about 1500 new digest mails are loaded into the queue.
I’m not sure if I should keep flushing this queue now. I also do not understand why, since the migration, there have been 100kish messages, given the forum had around 10k users.
Well, for me it was the same but with SMF and with over 750k enqueued jobs and growing even though I set it up to 50 threads… I guess the nuclear option worked.