Sv hup unicorn dropping traffic

Support
1

I’ve been running into a ~40s downtime when reloading my site. It’s a bit beyond my technical abilities, so I’ve asked Claude for help.

It claims the problem is that the unicorn launcher is sending the wrong signal to the old master (SIGTERM instead of SIGQUIT), which kills it before the new one can start serving.

The fix is apparently a 6-line update, but I’d like to sanity check it here first.

Click to see the Claude-generated report

Summary

On the pitchfork code path, config/unicorn_launcher’s on_reload_pitchfork() retires the outgoing master as soon as a worker process exists, without ever checking that the new master can serve a request, and it retires it with kill (SIGTERM, immediate shutdown) rather than kill -s QUIT (graceful shutdown).

The unicorn path immediately above it, on_reload_unicorn(), does both correctly — it issues curl $LOCAL_WEB to warm the new master and then sends QUIT.

The observed result of a single sv hup unicorn is ~40 seconds during which no request completes, ending exactly when the new generation logs Booted Rails.

Environment

  • Discourse 2026.8.0-latest.1, Rails 8.0.5.1, Ruby 3.4.10
  • pitchfork 0.18.2, RUN_PITCHFORK unset (so the pitchfork branch is active)
  • standalone container, nginx → http://127.0.0.1:3000
  • 2 vCPU / 4 GB droplet; app boot takes ~40s

The code

config/unicorn_launcher:

function on_reload_unicorn() {
  ...
  while [ ... workers not up ... ]; do sleep 1; done

  curl $LOCAL_WEB &>/dev/null      # warm the new master
  kill -s QUIT $UNICORN_PID        # graceful
}

function on_reload_pitchfork() {
  log "Reloading pitchfork ($UNICORN_PID)"
  OLD_PID=$UNICORN_PID
  pitchfork "${PITCHFORK_ARGS[@]}" &
  UNICORN_PID=$!

  count=0
  while [ "$count" -lt 180 -a -z "$(pgrep -f -P $UNICORN_PID worker)" ]; do
    log "Waiting for new pitchfork workers under $UNICORN_PID to start up..."
    count=$((count + 1))
    sleep 1
  done

  kill $OLD_PID 2>/dev/null        # SIGTERM, and no warm-up first
}

pgrep -f -P $UNICORN_PID worker returns as soon as a worker process is spawned. On pitchfork the worker exists long before the application has finished loading, so the loop exits early and the old master is torn down while the new one is still booting.

Observed behaviour

production.log, one reload (timestamps UTC):

23:16:49  Completed 200 OK in 95ms        <- outgoing generation, warm
          (44 seconds; not one Completed line)
23:17:33  Booted Rails 8.0.5.1 application in production environment
23:17:36  Completed 200 OK in 4128ms      <- first render on the new generation, cold
23:17:39  Completed 200 OK in 200ms

A monitor polling GET / every ~2s through nginx during that window recorded:

23:17:11  499  19.697s   (client gave up; upstream never answered)
23:17:29  503  16.301s
23:17:31  502   1.448s

and nginx’s error log for the 502:

2026/08/14 23:17:31 [error] 70#70: *110675 recv() failed (104: Connection reset by peer)
while reading response header from upstream, upstream: "http://127.0.0.1:3000/"

upstream_response_time equals request_time in every failing line, so nginx is not the bottleneck — the requests reach a process that cannot answer them. The reset at 23:17:31 is consistent with SIGTERM tearing down the outgoing master with connections still queued on it.

This is reproducible: two reloads in one deployment produced two identical ~40s windows, each ending at a Booted Rails line.

Not resource pressure: no OOM kills in dmesg, swap at 141 MB of 2048 used, load average 0.07, and both boots completed normally.

Suggested fix

Mirror the unicorn branch — wait until the new master actually serves before retiring the old one, and retire it gracefully:

   count=0
   while [ "$count" -lt 180 -a -z "$(pgrep -f -P $UNICORN_PID worker)" ]; do
     log "Waiting for new pitchfork workers under $UNICORN_PID to start up..."
     count=$((count + 1))
     sleep 1
   done

-  kill $OLD_PID 2>/dev/null
+  count=0
+  until curl -sf -o /dev/null "$LOCAL_WEB" || [ "$count" -ge 180 ]; do
+    log "Waiting for the new pitchfork master to serve a request..."
+    count=$((count + 1))
+    sleep 1
+  done
+
+  kill -s QUIT $OLD_PID 2>/dev/null

The QUIT-instead-of-TERM half is unambiguous: pitchfork inherits unicorn’s signal semantics, where QUIT drains and TERM is the immediate stop.

PS: This is my first AI-assisted report. Let me know if I’m breaking etiquette in any way. :sweat_smile: