Transient 503s after ./launcher rebuild appN when they’re fronted by HAProxy

Product Support Self-hosting
,
4

For anyone running multiple web containers, one extra piece to think about is where Sidekiq runs.

503s only affect web requests, but if Sidekiq is down your site will quietly stop handling background jobs (emails, digests, badges, webhooks). A neat way to handle this is to give Sidekiq its own container and keep it disabled on the web nodes.

Discourse uses runit under the hood: each service in /etc/service/* starts unless there’s a file named down inside its directory. That’s all the control that’s needed.

Here’s an example of a dedicated Sidekiq worker:

worker.yml — Sidekiq only 
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/sshd.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"

expose: []   # no HTTP ports

env:
  DISCOURSE_DB_HOST: data
  DISCOURSE_REDIS_HOST: data
  DISCOURSE_HOSTNAME: "forum.example.com"
  # match your existing secrets / SMTP settings

hooks:
  after_code:
    - exec:
        cd: $home
        cmd:
          # disable web services
          - mkdir -p /etc/service/puma  && touch /etc/service/puma/down
          - mkdir -p /etc/service/nginx && touch /etc/service/nginx/down
          # enable Sidekiq
          - rm -f /etc/service/sidekiq/down

run:
  - exec: echo "Starting Sidekiq worker container"

On the web containers (app1/app2), the opposite should be done - keep Puma and Nginx enabled, but add:

hooks:
  after_code:
    - exec:
        cd: $home
        cmd:
          - mkdir -p /etc/service/sidekiq && touch /etc/service/sidekiq/down

With this setup, HAProxy balances only the web containers, while Sidekiq keeps running in the worker container (.\launcher rebuild <yml name>). That way, background jobs aren’t interrupted when you rebuild or rotate your web nodes.