Delayed jobs are intermittently lost between the Sidekiq schedule zset and execution (no perform, no retry, no trace)

Contribute Bug

AI-generated summary

StephaneBRT reports a critical bug in Discourse 2026.10.0-latest.1 (Sidekiq 7.3.10, Ruby 4.0.9, Rails 8.1.4) where delayed jobs enqueued via Jobs.enqueue_in are intermittently lost. Approximately 40% of these jobs vanish from the Sidekiq schedule zset at their execution time without being processed, retried, or logged in stat:failed or Logster. The issue emerged after upgrading from Discourse 2026.2.0 (Ruby 3.3.8), which handled the same jobs reliably.

The reporter details extensive diagnostics showing:

  • The Sidekiq process remains healthy with zero restarts and no busy threads.
  • Database monitors confirm no SQL queries were executed for the lost jobs.
  • No exceptions or log entries exist in Logster::RedisStore.
  • The affected job (Jobs::NotifyMailingListSubscribers) is identical in class and payload for both lost and successful executions.

As a temporary mitigation, the user set email_time_window_mins = 0, forcing jobs to enqueue immediately rather than as delayed tasks, which bypasses the broken scheduling mechanism. This resolved the data loss but changed the notification batching behavior.

The post raises two key questions for the development team:

  1. Is this a known regression associated with the specific combination of Discourse 2026.10, Sidekiq 7.3.10, Ruby 4.0, or the Demon::Sidekiq fork model?
  2. What specific diagnostic data (e.g., DISCOURSE_LOG_SIDEKIQ, thread dumps) should be captured to aid in troubleshooting future occurrences?
1

Discourse 2026.10.0-latest.1 / Sidekiq 7.3.10 / Ruby 4.0.9

Summary

Since upgrading a self-hosted forum from Discourse 2026.2.0-latest (Ruby 3.3.8) to 2026.10.0-latest.1 (Ruby 4.0.9, Rails 8.1.4, Sidekiq 7.3.10) on 2026-10-26, jobs enqueued with Jobs.enqueue_in are intermittently and silently lost: the job disappears from the schedule zset at its due time but perform never runs. There is no retry, no dead job, no logged exception (we use Logster::RedisStore, so swallowed exceptions are visible — there are none), and stat:failed stays 0. Observed loss rate across 9 controlled/real occurrences: 4 lost / 5 executed (~40 %).

Environment

  • Official docker install (./launcher rebuild), single site
  • Discourse 2026.10.0-latest.1, image discourse/base:2.0.20261026-0138
  • Ruby 4.0.9 (2026-10-23), Rails 8.1.4, Sidekiq 7.3.10
  • UNICORN_SIDEKIQS=1 (sidekiq supervised by Demon::Sidekiq, forked by pitchfork)
  • Sidekiq state in a dedicated Redis database (Discourse.sidekiq_redis_config), separate from the Discourse redis
  • Sidekiq process healthy during all observations: 0 restarts, [0 of 5 busy], queues empty, retry = 0, dead = 0, stat:failed = 0
  • No OOM, no dmesg events, no container restarts

Observations (2026-10-30, real-world case + controlled experiments)

The affected job is Jobs::NotifyMailingListSubscribers, enqueued with Jobs.enqueue_in(SiteSetting.email_time_window_mins.minutes, ...) (600 s) → schedule zset with score = due time.

Due time (UTC) Job Outcome
15:33:05 real (post 282) lost — gone from schedule, never performed
15:50:10 control (post_id 999999 → silent early-return) executed (+1 stat:processed ~8 s after due)
16:04:18 real (post 283) executed (SMTP errors written, see below)
16:21:44 real ×2 + control all executed
17:34:39 real ×2 lost ×2
17:42:39 control executed (+1, 11 s after due)
17:55:59 real lost

Evidence gathered for each “lost” job:

  • zrange schedule no longer contains the payload (the zrem happened), and it is present in neither retry, nor dead, nor any queue (queue:low et al. all 0), nor stat:processed attribution.
  • A pg_stat_activity watcher sampling every 300 ms across the due window recorded zero SQL from the Sidekiq connection — not even the initial SELECT ... FROM posts that execute performs first.
  • Logster::RedisStore (__LOGSTER__MAP) contains no new entry for these jobs, while entries ARE written for the executed runs (SMTP failures of a different, unrelated issue we fixed in parallel).
  • Sidekiq proctitle stays [0 of 5 busy]; no sidekiq restart (same PID since boot), no OOM.
  • SkippedEmailLog count unchanged for lost jobs (the executed-and-failing runs do write rows there).

The “executed” runs are positively confirmed: they produced per-user SMTP error entries (captured in SkippedEmailLog.custom_reason), so the job class and payload shape are identical between lost and executed occurrences (same class, same queue low, only post_id differs).

The previous installation (Discourse 2026.2.0-latest, Ruby 3.3.8) did not exhibit this behaviour: the same delayed mailing-list jobs were delivered reliably on 2026-10-01 and 2026-10-21 (verified via EmailLog + downstream MTA queues).

Mitigation applied

We set email_time_window_mins = 0 for this site so that enqueue_in(0) skips the at key and pushes the job directly to its queue (client_push without at), bypassing the scheduled poller entirely. After this change, all 6 expected EmailLog rows were created immediately and the mails were accepted by the downstream MTA.

Trade-off accepted: mailing-list notifications are now sent immediately instead of batched +10 min.

Questions

  1. Is this a known regression in the 2026.10 / Sidekiq 7.3.10 combination (or in Demon::Sidekiq / the scheduled poller under Ruby 4.0)?
  2. Which additional diagnostics would you like us to capture on the next occurrence (e.g. enabling DISCOURSE_LOG_SIDEKIQ=1 job instrumentation, Sidekiq exception backtraces, poller thread dumps)?
2 Likes