Site settings are stale in Sidekiq after it re-forks

Contribute Bug
1

Priority: Medium, no data loss, but it silently sends user-facing PMs from the wrong
account, and any site setting changed after boot can be wrong inside background jobs.

Platform: Self-hosted, standard discourse_docker two-container setup (data +
web_only). Core 2026.7.0-latest (30d8364f0ab). Not browser-specific — server-side.

Description

Actual result: After the Sidekiq demon re-forks (e.g. after the RSS memory restart in
Demon::Sidekiq.rss_memory_check), the new Sidekiq process serves site settings from the
unicorn master’s boot-time snapshot, not from the database. Any setting changed since the
master booted is silently wrong inside background jobs, until either (a) the setting is
changed again while that Sidekiq process is alive, or (b) the container is restarted.

The visible symptom on my site: automated system PMs (post_hidden, flags_agreed_and_post_deleted, auto-unsuspend) were sent from a former value of
site_contact_username. The database had held the corrected value for five days, and the
staff action log showed no writes in that window. Meanwhile the exact same message types sent from a web request used the correct user, because web workers had processed the MessageBus refresh and Sidekiq had not.

Expected result: A freshly forked demon should read current site settings. SiteSetting
values in Sidekiq should match the database, regardless of how many times Sidekiq has
restarted since the master booted.

Reproducible steps

  1. Boot a self-hosted instance. Note the current site_contact_username (call it userA).
  2. In Admin → Settings, change site_contact_username to userB. All running processes
    pick this up correctly (MessageBus /site_settings → SiteSetting.refresh!).
  3. Kill the Sidekiq process inside the container so the unicorn master re-forks it
    (kill <sidekiq_pid>; or simply wait for Demon::Sidekiq.rss_memory_check to restart it once it passes the 1000 MB RSS threshold — on a busy site this happens on its own).
  4. Trigger any system message that is delivered via Jobs::SendSystemMessage — e.g. let a post be hidden by community flags, which enqueues :send_system_message from Post#hide!.
  5. Open the resulting PM.

Observed: The PM is authored by userA — the value that was current when the master
booted, not userB.
Expected: authored by userB.

Analysis

Discourse.after_fork calls SiteSetting.after_fork (lib/site_setting_extension.rb:736):

def after_fork
  @process_id = nil
  ensure_listen_for_changes
end

It never calls refresh!, so the child inherits the parent’s current settings hash via
copy-on-write and keeps it for its whole lifetime unless a subsequent change is broadcast.

ensure_listen_for_changes (lib/site_setting_extension.rb:713) is also effectively a no-op in the child, because @subscribed is inherited as true:

def ensure_listen_for_changes
  return if @listen_for_changes == false

  unless @subscribed
    MessageBus.subscribe(SITE_SETTINGS_CHANNEL) { |message| ... }
    @subscribed = true
  end
end

It works at all only because MessageBus.after_fork runs first (lib/discourse.rb:1064) and revives the bus thread against the inherited callback registry.

The demon is a plain fork from the master (lib/demon/base.rb:181), and Sidekiq re-forks
routinely in production: Demon::Sidekiq.rss_memory_check restarts it above
DEFAULT_MAX_ALLOWED_SIDEKIQ_RSS_MEGABYTES = 1000. Each fork resurrecting a value that had last been true two weeks earlier.

This is long-standing rather than a regression: after_fork has had this shape since
8fc2549 (2014), and is unchanged on current main.

Why it is easy to miss: at first boot, master and Sidekiq share the same (correct)
snapshot, and while Sidekiq stays alive it does receive change broadcasts. The divergence only appears for settings changed after master boot, in a Sidekiq process forked after that
change — i.e. it starts silently, some time later, with no error anywhere.

Impact beyond the contact user

site_contact_username is just the visible case, because it stamps a username onto a PM that users read. The same staleness applies to any site setting consulted inside a job — rate
limits, email/notification settings, feature toggles, plugin settings — with no error and no
log line. Operators reasonably conclude “the setting won’t stick” and go looking for something overwriting the database.

Suggested fix

In SiteSetting.after_fork, drop the inherited state and re-read:

def after_fork
  @process_id = nil
  @subscribed = false
  ensure_listen_for_changes
  refresh!
end

@subscribed = false also makes the child’s own subscription explicit rather than relying on MessageBus.after_fork reviving the inherited one.

Workaround for operators

./launcher restart web_only (a full container restart, so the master re-reads the setting at boot). Re-saving the setting in admin only fixes the currently alive Sidekiq process — the
next re-fork silently reverts it.

1 Like
3

Thank you for the report! I have opened a fix for this in

2 Likes