I'm using a Unix socket between Caddy and Discourse instead of TCP, but from time to time I get a 'unix:' error

Support
13

@Falco @satonotdead

I managed to fix it. I waited a good while to make sure it was really fixed, and it seems it is.

The setup

Discourse runs in Docker and exposes nginx on a Unix socket (/var/discourse/shared/standalone/nginx.http.sock). Caddy sits in front of it as a reverse proxy and passes the client’s real IP through X-Real-IP.

Two things had to change: the nginx config inside the container, and the way I run the rebuild.

1. app.yml

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  # This after_code block is just my own plugin list — it has nothing to do with
  # the fix. Keep whatever you already have here.
  after_code:
    - exec:
        [...]

  # This is the part that matters.
  #   1. Writes an http-level map that turns the literal "unix:" into 127.0.0.1.
  #      The 00- prefix makes nginx load it before discourse.conf.
  #   2. Rewrites discourse.conf so X-Forwarded-For uses the mapped variable
  #      instead of the raw $remote_addr.
  #   3. nginx -t fails the build if the result is not valid.
  after_web_config:
    - exec: >-
        printf 'map $remote_addr $safe_remote_addr {\n  "unix:" 127.0.0.1;\n  default $remote_addr;\n}\nreal_ip_header X-Real-IP;\n'
        > /etc/nginx/conf.d/00-safe-remote-addr.conf
    - exec: >-
        sed -i 's/X-Forwarded-For \$remote_addr;/X-Forwarded-For $safe_remote_addr;/g'
        /etc/nginx/conf.d/discourse.conf
    - exec: nginx -t

The critical code is what sits inside the after_web_config body.

2. The Caddyfile block

The script below drives maintenance mode through a flag file, so your site block needs to know about it:

forum.example.com {
    request_header X-Real-IP {remote_host}

    root * /var/caddy/flags
    @maintenance file maintenance.flag
    handle @maintenance {
        respond "Maintenance in progress. We'll be back in a few minutes." 503
    }

    handle {
        reverse_proxy unix//var/discourse/shared/standalone/nginx.http.sock
    }
}

The flag path in the script and the file the @maintenance matcher looks for have to be the same file. If you rename one, rename the other, otherwise maintenance mode silently never engages.

If the same Caddy instance also serves other apps, use a flag that only the forum’s block matches, otherwise a Discourse rebuild takes those other apps down with it.

3. The rebuild script

I no longer use ./launcher rebuild app directly. I use discourse-rebuild.sh:

#!/bin/bash
#
# discourse-rebuild.sh — Rebuilds the Discourse container without leaving orphaned
# requests behind and without the `invalid input syntax for type inet: "unix:"` error.
#
# CONTEXT
#   Discourse runs in Docker and exposes nginx on a Unix socket
#   (/var/discourse/shared/standalone/nginx.http.sock). Caddy acts as a reverse
#   proxy in front of it and passes the client's real IP through X-Real-IP.
#
#   A `rebuild` destroys the container and recreates the socket with a new inode.
#   During that transition there are two problematic windows:
#
#     1) Caddy keeps state from the previous socket until it is reloaded.
#     2) nginx starts accepting connections as soon as it boots, but Unicorn
#        takes ~15s longer before it can serve them.
#
#   A request landing in either window may arrive with no X-Real-IP. $remote_addr
#   is then left holding the literal "unix:", which PostgreSQL rejects when
#   inserting it into an inet column -> HTTP 500.
#
# WHAT IT DOES
#   1. Raises a flag file that puts the site into 503 (Caddy checks it on every
#      request, so it takes effect instantly and with no reload).
#   2. Waits until nothing but nginx itself is holding the socket open.
#   3. Rebuilds the container.
#   4. Polls /srv/status against the socket until Unicorn answers 200.
#   5. Reloads Caddy so it picks up the new socket, and clears the flag.
#   6. Starts the watcher that logs any leftover "unix:" hit.
#
#   If the rebuild fails, or Discourse never answers within the polling window,
#   the flag is NOT removed: the site stays in maintenance on purpose, so a broken
#   container is never exposed. Bring it back up by hand with:
#       rm -f /var/caddy/flags/maintenance.flag
#
# REQUIREMENTS
#   - lsof installed, and root privileges.
#   - FLAG below must point at the exact same file the Caddyfile @maintenance
#     matcher looks for.
#   - request_header X-Real-IP {remote_host} in that same Caddyfile block.
#
# USAGE
#   ./discourse-rebuild.sh
#
# TO SEE WHAT THE WATCHER CAUGHT
#   cat /var/log/unixip-hits.log
#

set -e

FLAG=/var/caddy/flags/maintenance.flag
SOCK=/var/discourse/shared/standalone/nginx.http.sock

cleanup() {
  local code=$?
  systemctl reload caddy
  if [ $code -eq 0 ]; then
    rm -f "$FLAG"
    echo "✅ Rebuild finished. Site is online."
  else
    echo "⚠️  Rebuild failed (exit code $code). The site is still in maintenance."
    echo "    Check it, and once it's ready: rm -f $FLAG"
  fi
}
trap cleanup EXIT

mkdir -p "$(dirname "$FLAG")"
touch "$FLAG"
echo "🔧 Maintenance is on. Waiting for in-flight requests..."

# Rough heuristic: count the processes holding the socket open and wait until
# only the listener is left. Up to 30s.
for i in $(seq 30); do
  n=$(lsof -t "$SOCK" 2>/dev/null | wc -l || echo 0)
  [ "$n" -le 1 ] && break
  sleep 1
done

/var/discourse/launcher rebuild app

# Up to 60 attempts: about 2 minutes of sleeps, more if any curl hits its own
# 5s timeout.
echo "⏳ Waiting for Discourse to answer..."
status=000
for i in $(seq 60); do
  status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
    --unix-socket "$SOCK" http://localhost/srv/status 2>/dev/null || echo 000)
  [ "$status" = "200" ] && break
  sleep 2
done

if [ "$status" != "200" ]; then
  echo "⚠️  Discourse never answered within the polling window (last status code: $status)."
  exit 1
fi

echo "✅ Discourse ready after ~$((i*2))s."

systemd-run --unit=unixip-watch --collect \
  /bin/bash -c "docker exec app tail -F /var/log/nginx/access.log | grep --line-buffered 'unix:' >> /var/log/unixip-hits.log"

What the script actually does

  1. Puts the site into maintenance through Caddy’s flag file.
  2. Waits for the socket to go quiet, then runs ./launcher rebuild app.
  3. Polls /srv/status over the socket until Unicorn answers 200. This is the critical part: it is what stops requests from reaching nginx while Unicorn is still booting.
  4. Reloads Caddy (belt and braces, it did not turn out to be critical).
  5. Clears the maintenance flag only if that poll succeeded.

Several weeks and several rebuilds later, I have not seen that error again.

PS: I really don’t know what the hell I did.