I’ve got a high-volume system with 24GB ram, a 30GB on disk database, and 2M page views/month. There were some bumps as the site got migrated and updated to the new image format, but last I looked it appeared to be running smoothly according to those familiar with the site. Earlier today I noticed 25K sidekiq jobs. They got processed pretty quickly, but sidekiq reported a 10Ks latency. A user reported that they had received 5 summary emails in the space of an hour. Here’s the mail-sent graph: [image] Here’s grafana stats for the day. [image] The site was upgraded a couple days back when I installed the prometheus exporter plugin. There were complaints of multiple summary emails earlier, but I chalked that up to the site being swamped due to the image processing. Does anyone have an explanation for why a user might receive multiple summary emails in a day?

Database corruption, maybe? We have no other systemic reports of digest emails going out inappropriately. Did you check and make sure this user doesn’t have anything weird in their user settings for the activity summary (aka “digest”) [image]

The ones I inspected had “weekly” checked. I can have a closer look at the rails console. Any other idea of what kind of corruption could cause this? .

Zero idea but we have no other reports of this, so it’s specific to the migration or config somehow. Could there be multiple instances running or multiple sidekiqs or anything bizarre like that? :man_shrugging:

Something like multiple sidekiq is something I considered, or somehow sidekiq is slow so it (or postgres) doesn’t report that the digest has been created when it checks again? Remember that we were concerned about disk performance. Is it possible that postgres could somehow not commit that the digest has been sent and another one gets started before the last one gets finished? I’ll double check that there isn’t somehow an extra container running somehow. EDIT: I see just the data, web, and mail-receiver containers. I checked the mail logs for the user that complained. There were four digests yesterday: created_at: Thu, 21 Mar 2019 15:11:33 UTC +00:00, created_at: Thu, 21 Mar 2019 15:58:46 UTC +00:00, created_at: Thu, 21 Mar 2019 17:15:29 UTC +00:00, created_at: Thu, 21 Mar 2019 18:35:39 UTC +00:00, Each took had the same time for created_at as updated_at. UserOptions look normal. digest_after_minutes: 10080,

I think I’ve identified a race condition. I know that the sidekiq queue had a 10K second (~3 hour) latency. Messages are queued every 30 minutes: app/jobs/scheduled/enqueue_digest_emails.rb main module Jobs class EnqueueDigestEmails < ::Jobs::Scheduled So a digest gets queued in an hour-long queue. Half an hour later we test for whether to queue another digest, checking the last time that a message was sent: app/jobs/scheduled/enqueue_digest_emails.rb main target_user_ids.each do |user_id| ::Jobs.enqueue(:user_email, type: "digest", user_id: user_id) end end def target_user_ids # Users who want to receive digest email within their chosen digest email frequency query = User .real .activated .not_staged .not_suspended .joins(:user_option, :user_stat, :user_emails) .where("user_options.email_digests") .where( "COALESCE(user_options.digest_after_minutes, ?) > 0", SiteSetting.default_email_digest_frequency, ) .where("user_stats.bounce_score < ?", SiteSetting.bounce_score_threshold) .where("user_emails.primary") The last digest is queued, but not sent, so last_emailed_at isn’t updated, so another message gets queued. The solutions that I see are: this site gets faster hardware (beyond my control) target_user_ids somehow searches the queue to see if there is a queued, unsent digest (seems really hard and won’t get into core in my lifetime). Have the process that sends the digest do another check of last_emailed_at before sending the digest (I think this might be the best long-term solution, but haven’t found that code) I add a hook to modify enqueue_digest_emails.rb to make every 30.minutes something like 4.hours (6 hours?) and hope that is enough time for the queue to get processed before it runs again. Number 4 above seems like my only immediate solution. Is there some deleterious side effect that I am overlooking? (I did recently increase db_shared_buffers from 10GB to 16GB for the 30GB on disk database. This might help, but I suspect that it won’t be enough. There are ~25K users at TL1 and above.)

Entirely plausible, I suggest running basic disk perf stats, if your disks are slow (non-SSD) and you have a lot of activity and a large database.. it isn’t going to be pretty.

We both know that the disk performance is insufficient. That is beyond my control. I guess I’ll up the digest queue to 4 hours. I don’t see an alternative.

[image] pfaffman: Have the process that sends the digest do another check of last_emailed_at before sending the digest (I think this might be the best long-term solution, but haven’t found that code) I think this is a good solution too. Summary emails (and many others) are put into the low-priority queue, so are most likely to be delayed when sidekiq has a lot of jobs to process. I don’t think it’s too much of a hack to check last_emailed_at if it’s a digest email in here.

Thanks, @neil. So something like return if type.to_s == "digest" && user.last_emailed_at < user.digest_after_minutes right about app/jobs/regular/user_email.rb main end def set_skip_context(type, user_id, to_address, post_id) @skip_context = { type: type, user_id: user_id, to_address: to_address, post_id: post_id } end NOTIFICATIONS_SENT_BY_MAILING_LIST = Set.new %w[posted replied mentioned group_mentioned quoted] def message_for_email(user, post, type, notification, args = nil) args ||= {} notification_type = args[:notification_type] notification_data_hash = args[:notification_data_hash] email_token = args[:email_token] to_address = args[:to_address] set_skip_context(type, user.id, to_address || user.email, post.try(:id)) if user.anonymous? return skip_message(SkippedEmailLog.reason_types[:user_email_anonymous_user]) end ?

Yup, that’s about right. return if type.to_s == "digest" && user.last_emailed_at < user.user_option.digest_after_minutes.minutes.ago

Users receive multiple summary emails when sidekiq queue is >30 minutes

Product Support Self-hosting

neil (Neil Lalonde) March 22, 2019, 4:35pm 21

Fix is here:

https://github.com/discourse/discourse/commit/399e937a38f4b4917e0c44897c308d39d4a844e1

Topic		Replies	Views
Sidekiq Error Many Summary Mail sent failed Product Support	0	407	November 15, 2021
Digests all sending at the same time Product Support activity-summary	5	971	August 23, 2018
More then 3 million enqueued jobs show in sidekiq? How to I stop it generate the queue? Product Support	16	593	February 4, 2025
Cannot Send Digest Emails: Jobs::UserEmail Deprecation Notice / No Emails After Enqueue via Rails Product Support email , activity-summary	5	220	November 3, 2025
Posts in rapid succession do not send all emails Product Support	1	474	November 29, 2021

Users receive multiple summary emails when sidekiq queue is >30 minutes

Related topics