I'm using a Unix socket between Caddy and Discourse instead of TCP, but from time to time I get a 'unix:' error

תמיכה
1

The logs show the following:

PG::InvalidTextRepresentation (ERROR: invalid input syntax for type inet: "unix:" LINE 7: client_ip = 'unix:', ^ ) lib/mini_sql_multisite_connection.rb:109:in 'MiniSqlMult

rack-mini-profiler (4.0.1) lib/patches/db/pg/alias_method.rb:109:in 'PG::Connection#exec'
rack-mini-profiler (4.0.1) lib/patches/db/pg/alias_method.rb:109:in 'PG::Connection#async_exec'
mini_sql (1.6.0) lib/mini_sql/postgres/connection.rb:217:in 'MiniSql::Postgres::Connection#run'
mini_sql (1.6.0) lib/mini_sql/active_record_postgres/connection.rb:38:in 'block in MiniSql::ActiveRecordPostgres::Connection#run'
mini_sql (1.6.0) lib/mini_sql/active_record_postgres/connection.rb:34:in 'block in MiniSql::ActiveRecordPostgres::Connection#with_lock'
activesupport (8.0.5.1) lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
mini_sql (1.6.0) lib/mini_sql/active_record_postgres/connection.rb:34:in 'MiniSql::ActiveRecordPostgres::Connection#with_lock'
mini_sql (1.6.0) lib/mini_sql/active_record_postgres/connection.rb:38:in 'MiniSql::ActiveRecordPostgres::Connection#run'
lib/mini_sql_multisite_connection.rb:109:in 'MiniSqlMultisiteConnection#run'
mini_sql (1.6.0) lib/mini_sql/postgres/connection.rb:196:in 'MiniSql::Postgres::Connection#exec'
app/models/user_auth_token.rb:244:in 'UserAuthToken#rotate!'
lib/auth/default_current_user_provider.rb:260:in 'Auth::DefaultCurrentUserProvider#refresh_session'
lib/current_user.rb:52:in 'CurrentUser#refresh_session'
app/controllers/application_controller.rb:91:in 'ApplicationController#perform_refresh_session'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:361:in 'block in ActiveSupport::Callbacks::CallTemplate::MethodCall#make_lambda'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:207:in 'ActiveSupport::Callbacks::Filters::After#call'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:563:in 'block in ActiveSupport::Callbacks::CallbackSequence#invoke_after'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:563:in 'Array#each'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:563:in 'ActiveSupport::Callbacks::CallbackSequence#invoke_after'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:134:in 'block in ActiveSupport::Callbacks#run_callbacks'
app/controllers/application_controller.rb:444:in 'block in ApplicationController#with_resolved_locale'
i18n (1.14.8) lib/i18n.rb:354:in 'I18n::Base#with_locale'
app/controllers/application_controller.rb:444:in 'ApplicationController#with_resolved_locale'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
app/controllers/application_controller.rb:1126:in 'ApplicationController#ensure_dont_cache_page'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:129:in 'block in ActiveSupport::Callbacks#run_callbacks'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:140:in 'ActiveSupport::Callbacks#run_callbacks'
actionpack (8.0.5.1) lib/abstract_controller/callbacks.rb:260:in 'AbstractController::Callbacks#process_action'
actionpack (8.0.5.1) lib/action_controller/metal/rescue.rb:27:in 'ActionController::Rescue#process_action'
actionpack (8.0.5.1) lib/action_controller/metal/instrumentation.rb:76:in 'block in ActionController::Instrumentation#process_action'
activesupport (8.0.5.1) lib/active_support/notifications.rb:210:in 'block in ActiveSupport::Notifications.instrument'
activesupport (8.0.5.1) lib/active_support/notifications/instrumenter.rb:58:in 'ActiveSupport::Notifications::Instrumenter#instrument'
activesupport (8.0.5.1) lib/active_support/notifications.rb:210:in 'ActiveSupport::Notifications.instrument'
actionpack (8.0.5.1) lib/action_controller/metal/instrumentation.rb:75:in 'ActionController::Instrumentation#process_action'
actionpack (8.0.5.1) lib/action_controller/metal/params_wrapper.rb:259:in 'ActionController::ParamsWrapper#process_action'
activerecord (8.0.5.1) lib/active_record/railties/controller_runtime.rb:39:in 'ActiveRecord::Railties::ControllerRuntime#process_action'
actionpack (8.0.5.1) lib/abstract_controller/base.rb:152:in 'AbstractController::Base#process'
actionview (8.0.5.1) lib/action_view/rendering.rb:40:in 'ActionView::Rendering#process'
rack-mini-profiler (4.0.1) lib/mini_profiler/profiling_methods.rb:90:in 'block in ActionController::Base#profile_method'
actionpack (8.0.5.1) lib/action_controller/metal.rb:252:in 'ActionController::Metal#dispatch'
actionpack (8.0.5.1) lib/action_controller/metal.rb:335:in 'ActionController::Metal.dispatch'
actionpack (8.0.5.1) lib/action_dispatch/routing/route_set.rb:67:in 'ActionDispatch::Routing::RouteSet::Dispatcher#dispatch'
actionpack (8.0.5.1) lib/action_dispatch/routing/route_set.rb:50:in 'ActionDispatch::Routing::RouteSet::Dispatcher#serve'
actionpack (8.0.5.1) lib/action_dispatch/journey/router.rb:53:in 'block in ActionDispatch::Journey::Router#serve'
actionpack (8.0.5.1) lib/action_dispatch/journey/router.rb:133:in 'block in ActionDispatch::Journey::Router#find_routes'
actionpack (8.0.5.1) lib/action_dispatch/journey/router.rb:126:in 'Array#each'
actionpack (8.0.5.1) lib/action_dispatch/journey/router.rb:126:in 'ActionDispatch::Journey::Router#find_routes'
actionpack (8.0.5.1) lib/action_dispatch/journey/router.rb:34:in 'ActionDispatch::Journey::Router#serve'
actionpack (8.0.5.1) lib/action_dispatch/routing/route_set.rb:908:in 'ActionDispatch::Routing::RouteSet#call'
lib/middleware/omniauth_bypass_middleware.rb:35:in 'Middleware::OmniauthBypassMiddleware#call'
lib/middleware/crawler_hooks.rb:13:in 'Middleware::CrawlerHooks#call'
rack (2.2.23) lib/rack/tempfile_reaper.rb:15:in 'Rack::TempfileReaper#call'
rack (2.2.23) lib/rack/conditional_get.rb:27:in 'Rack::ConditionalGet#call'
rack (2.2.23) lib/rack/head.rb:12:in 'Rack::Head#call'
actionpack (8.0.5.1) lib/action_dispatch/http/permissions_policy.rb:38:in 'ActionDispatch::PermissionsPolicy::Middleware#call'
lib/content_security_policy/middleware.rb:12:in 'ContentSecurityPolicy::Middleware#call'
lib/middleware/anonymous_cache.rb:425:in 'Middleware::AnonymousCache#call'
lib/middleware/csp_script_nonce_injector.rb:13:in 'Middleware::CspScriptNonceInjector#call'
lib/middleware/track_view_session_id_injector.rb:12:in 'Middleware::TrackViewSessionIdInjector#call'
config/initializers/008-rack-cors.rb:14:in 'Discourse::Cors#call'
rack (2.2.23) lib/rack/session/abstract/id.rb:266:in 'Rack::Session::Abstract::Persisted#context'
rack (2.2.23) lib/rack/session/abstract/id.rb:260:in 'Rack::Session::Abstract::Persisted#call'
actionpack (8.0.5.1) lib/action_dispatch/middleware/cookies.rb:706:in 'ActionDispatch::Cookies#call'
actionpack (8.0.5.1) lib/action_dispatch/middleware/callbacks.rb:31:in 'block in ActionDispatch::Callbacks#call'
activesupport (8.0.5.1) lib/active_support/callbacks.rb:100:in 'ActiveSupport::Callbacks#run_callbacks'
actionpack (8.0.5.1) lib/action_dispatch/middleware/callbacks.rb:30:in 'ActionDispatch::Callbacks#call'
actionpack (8.0.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:31:in 'ActionDispatch::DebugExceptions#call'
actionpack (8.0.5.1) lib/action_dispatch/middleware/show_exceptions.rb:32:in 'ActionDispatch::ShowExceptions#call'
logster (2.21.0) lib/logster/middleware/reporter.rb:40:in 'Logster::Middleware::Reporter#call'
lib/middleware/default_headers.rb:13:in 'Middleware::DefaultHeaders#call'
railties (8.0.5.1) lib/rails/rack/logger.rb:41:in 'Rails::Rack::Logger#call_app'
railties (8.0.5.1) lib/rails/rack/logger.rb:29:in 'Rails::Rack::Logger#call'
config/initializers/100-quiet_logger.rb:20:in 'DiscourseRackQuietAssetsLogger#call'
config/initializers/100-silence_logger.rb:29:in 'SilenceLogger#call'
actionpack (8.0.5.1) lib/action_dispatch/middleware/request_id.rb:34:in 'ActionDispatch::RequestId#call'
lib/middleware/enforce_hostname.rb:23:in 'Middleware::EnforceHostname#call'
rack (2.2.23) lib/rack/method_override.rb:24:in 'Rack::Metho

Job exception: ERROR: invalid input syntax for type inet: "unix:" LINE 2: SET ip_address = 'unix:' ^

rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in 'PG::Connection#exec'
rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in 'PG::Connection#async_exec'
mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:217:in 'MiniSql::Postgres::Connection#run'
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in 'block in MiniSql::ActiveRecordPostgres::Connection#run'
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in 'block in MiniSql::ActiveRecordPostgres::Connection#with_lock'
activesupport-8.0.5.1/lib/active_support/concurrency/null_lock.rb:9:in 'ActiveSupport::Concurrency::NullLock#synchronize'
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in 'MiniSql::ActiveRecordPostgres::Connection#with_lock'
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in 'MiniSql::ActiveRecordPostgres::Connection#run'
/var/www/discourse/lib/mini_sql_multisite_connection.rb:109:in 'MiniSqlMultisiteConnection#run'
mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:196:in 'MiniSql::Postgres::Connection#exec'
/var/www/discourse/app/models/user.rb:1145:in 'User.update_ip_address!'
/var/www/discourse/lib/auth/default_current_user_provider.rb:235:in 'block in Auth::DefaultCurrentUserProvider#current_user'
/var/www/discourse/lib/scheduler/defer.rb:135:in 'block in Scheduler::Deferrable#do_work'
rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in 'RailsMultisite::ConnectionManagement::NullInstance#with_connection'
rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in 'RailsMultisite::ConnectionManagement.with_connection'
/var/www/discourse/lib/scheduler/defer.rb:130:in 'Scheduler::Deferrable#do_work'
/var/www/discourse/lib/scheduler/defer.rb:116:in 'block (2 levels) in Scheduler::Deferrable#start_thread'

When it happens, an “Oops - Error 500” page is displayed. At first, I thought Caddy was the culprit because it wasn’t forwarding the client’s IP address to Discourse, so I tried several different configurations, but none of them solved the issue.

It’s worth mentioning that this doesn’t happen very often. It occurs only occasionally, and when it does, simply refreshing the page immediately recovers the site. It then works normally for quite some time before the error randomly occurs again.

2

I’m not an expert and I did not use Caddy but Nginx on my self-hosted instance, but can I ask for your actual Caddy and Discourse templates configuration in ./app/containers.yml?

Can you remember what you usually did before the error happens? I mean, changing admin settings, posting, using some plugin?

I’m sorry not to give specific help to your issue, but I think your replies can add value to your question for getting a clear and fast response from the community.

3

This means that a request / user has a remote ip pointing to your socket, which means something in your proxy chain is misconfigured.

4

@satonotdead @Falco

app.yml

templates:
  - "templates/postgres.18.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  
  - "templates/web.socketed.template.yml"
  
  - "templates/enable-ruby-yjit.yml"


## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  # - "66:80"   # http
  # - "66:443" # https

params:
  ## Which Git revision should this container use? (default: latest)
  version: latest
  ## Maximum upload size (default: 10m)
  upload_size: 150m
  
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "2048MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## https://meta.discourse.org/t/rescaling-the-server-which-configs-need-to-be-changed-unicorn-workers-memory-etc/252788
  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 8

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: example.com

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'admin+discourse@example.com'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS is required
  # WARNING: SMTP password should be wrapped in quotes to avoid problems
  DISCOURSE_SMTP_ADDRESS: smtp.provider.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: noreply@example.com
  DISCOURSE_SMTP_PASSWORD: "***"
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default: true)
  DISCOURSE_SMTP_DOMAIN: example.com # (required by some providers)
  DISCOURSE_NOTIFICATION_EMAIL: noreply@example.com
  #DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: peer        # (optional, default: peer, valid values: none, peer, client_once, fail_if_no_peer_cert)
  #DISCOURSE_SMTP_AUTHENTICATION: plain            # (default: plain, valid values: plain, login, cram_md5)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  # LETSENCRYPT_ACCOUNT_EMAIL: admin+letsencrypt@example.com

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

  ## The maxmind geolocation IP account ID and license key for IP address lookups
  ## see https://meta.discourse.org/t/-/173941 for details
  #DISCOURSE_MAXMIND_ACCOUNT_ID: 123456
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

  # Force HTTPS
  DISCOURSE_FORCE_HTTPS: true
  
  # Request Limits
  DISCOURSE_MAX_REQS_PER_IP_MODE: none
  DISCOURSE_MAX_ADMIN_API_REQS_PER_MINUTE: 12000
  
  DISCOURSE_MAX_DATA_EXPLORER_API_REQ_MODE: none
  DISCOURSE_MAX_DATA_EXPLORER_API_REQS_PER_10_SECONDS: 1000

  DISCOURSE_YJIT_ENABLED: true

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log
  - volume:
      host: /var/discourse/plugins
      guest: /var/plugins

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - cp -a /var/plugins/. $home/plugins/

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"

Caddyfile

#---------------------------------------------------------------------------------

{
	storage file_system {
		root /var/caddy/data
	}
}

#---------------------------------------------------------------------------------

# To achieve an A+ rating in SSL security
(hsts) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
	}
}

# self-hosted acme-dns credentials
(tls-challenge) {
	tls admin+letsencrypt@example.com {
		dns acmedns {
			username ***
			password ***
			subdomain ***
			server_url http://acme.example.com:5005
		}
	}
}

#---------------------------------------------------------------------------------
# Forum (Discourse)
#---------------------------------------------------------------------------------
example.com {
	#-------------------------------------------------------------------------------
	import hsts
	import tls-challenge
	#-------------------------------------------------------------------------------
	request_header X-Real-IP {remote_host}
	reverse_proxy unix//var/discourse/shared/standalone/nginx.http.sock
	#-------------------------------------------------------------------------------
}

#---------------------------------------------------------------------------------
# Subdominios
#---------------------------------------------------------------------------------

#---------------------------------------------------------------------------------
*.example.com {
	#-------------------------------------------------------------------------------
	import hsts
	import tls-challenge
	#-------------------------------------------------------------------------------

	# Force the non-www version of the domain
	@www host www.example.com
	redir @www https://example.com{uri} permanent
  
	#-------------------------------------------------------------------------------
  
	@store host store.example.com
	handle @store {
		@wc_private {
			path /license.txt
			path /readme.html

			# Archivos críticos
			path /wp-config.php
			path /xmlrpc.php
			path /wp-settings.php
			path /wp-load.php
			path /wp-blog-header.php

			path /wp-admin.php

			path /wp-admin/install.php

			path /wp-content/uploads/wc-logs/*
			path /wp-content/uploads/nuvei-logs/*

			path /wp-content/uploads/woocommerce_uploads/*
		}

		# protect_php
		@wc_private_php {
			not path /wp-includes/ms-files.php

			path_regexp protect_php ^/(wp-includes|wp-admin/includes|wp-content/uploads)/.*\.php$
		}
		respond @wc_private 403
		respond @wc_private_php 403

		root * /usr/share/wordpress
		php_fastcgi unix//run/php/php8.3-fpm.sock
		file_server
	}
  
  #-------------------------------------------------------------------------------
  # Multiple standalone HTML files
  #-------------------------------------------------------------------------------

  @join host join.example.com
  handle @join {
    root * /var/example/join
    file_server
  }

  #-------------------------------------------------------------------------------
  # No matchers
  #-------------------------------------------------------------------------------

	handle {
		respond 404
	}

	#-------------------------------------------------------------------------------
}

To be more specific, this error occurs several hours after the previous one, it doesn’t happen very often. I don’t see anything unusual in app.yml or the Caddyfile.

5

If the connection to the socket is right until it gets dropped, I think you can add explicit timeouts and transport options.

6

I think you can add explicit timeouts and transport options

How?

7

Well, as I said before, I did not use Caddy but you can look into their official docs.

8

I think I found the problem. If I perform a rebuild, I never restart Caddy. From now on, I’m going to do this:

./launcher rebuild app && systemctl reload caddy

I suspect that Caddy uses the socket cache or something similar after a rebuild. But it doesn’t usually happen immediately after a rebuild. We’ll see.

9

I’ve been working on this for days. I’ve verified that Caddy sends the header, that nginx receives and applies it, that there’s no CDN, no other process touching the socket, and no webhooks. Every single test passes. And the Unix error keeps appearing. I thought it was because of the rebuild, but it’s not. All of this happens randomly over very long periods of time.

Is TCP my only option now?

10

Did you add the timeouts I suggested to your Caddy template? While I’m no expert, I think that might solve your problem.

11

Doing that doesn’t make sense because some requests fail in 15 seconds and others in less than 1 second. It’s basically pointless.

12

Right, you can check this two links:

It seems that you are missing a header and/or your chain into Discourse template file.

13

@Falco @satonotdead

I managed to fix it. I waited a good while to make sure it was really fixed, and it seems it is.

The setup

Discourse runs in Docker and exposes nginx on a Unix socket (/var/discourse/shared/standalone/nginx.http.sock). Caddy sits in front of it as a reverse proxy and passes the client’s real IP through X-Real-IP.

Two things had to change: the nginx config inside the container, and the way I run the rebuild.

1. app.yml

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  # This after_code block is just my own plugin list — it has nothing to do with
  # the fix. Keep whatever you already have here.
  after_code:
    - exec:
        [...]

  # This is the part that matters.
  #   1. Writes an http-level map that turns the literal "unix:" into 127.0.0.1.
  #      The 00- prefix makes nginx load it before discourse.conf.
  #   2. Rewrites discourse.conf so X-Forwarded-For uses the mapped variable
  #      instead of the raw $remote_addr.
  #   3. nginx -t fails the build if the result is not valid.
  after_web_config:
    - exec: >-
        printf 'map $remote_addr $safe_remote_addr {\n  "unix:" 127.0.0.1;\n  default $remote_addr;\n}\nreal_ip_header X-Real-IP;\n'
        > /etc/nginx/conf.d/00-safe-remote-addr.conf
    - exec: >-
        sed -i 's/X-Forwarded-For \$remote_addr;/X-Forwarded-For $safe_remote_addr;/g'
        /etc/nginx/conf.d/discourse.conf
    - exec: nginx -t

The critical code is what sits inside the after_web_config body.

2. The Caddyfile block

The script below drives maintenance mode through a flag file, so your site block needs to know about it:

forum.example.com {
    request_header X-Real-IP {remote_host}

    root * /var/caddy/flags
    @maintenance file maintenance.flag
    handle @maintenance {
        respond "Maintenance in progress. We'll be back in a few minutes." 503
    }

    handle {
        reverse_proxy unix//var/discourse/shared/standalone/nginx.http.sock
    }
}

The flag path in the script and the file the @maintenance matcher looks for have to be the same file. If you rename one, rename the other, otherwise maintenance mode silently never engages.

If the same Caddy instance also serves other apps, use a flag that only the forum’s block matches, otherwise a Discourse rebuild takes those other apps down with it.

3. The rebuild script

I no longer use ./launcher rebuild app directly. I use discourse-rebuild.sh:

#!/bin/bash
#
# discourse-rebuild.sh — Rebuilds the Discourse container without leaving orphaned
# requests behind and without the `invalid input syntax for type inet: "unix:"` error.
#
# CONTEXT
#   Discourse runs in Docker and exposes nginx on a Unix socket
#   (/var/discourse/shared/standalone/nginx.http.sock). Caddy acts as a reverse
#   proxy in front of it and passes the client's real IP through X-Real-IP.
#
#   A `rebuild` destroys the container and recreates the socket with a new inode.
#   During that transition there are two problematic windows:
#
#     1) Caddy keeps state from the previous socket until it is reloaded.
#     2) nginx starts accepting connections as soon as it boots, but Unicorn
#        takes ~15s longer before it can serve them.
#
#   A request landing in either window may arrive with no X-Real-IP. $remote_addr
#   is then left holding the literal "unix:", which PostgreSQL rejects when
#   inserting it into an inet column -> HTTP 500.
#
# WHAT IT DOES
#   1. Raises a flag file that puts the site into 503 (Caddy checks it on every
#      request, so it takes effect instantly and with no reload).
#   2. Waits until nothing but nginx itself is holding the socket open.
#   3. Rebuilds the container.
#   4. Polls /srv/status against the socket until Unicorn answers 200.
#   5. Reloads Caddy so it picks up the new socket, and clears the flag.
#   6. Starts the watcher that logs any leftover "unix:" hit.
#
#   If the rebuild fails, or Discourse never answers within the polling window,
#   the flag is NOT removed: the site stays in maintenance on purpose, so a broken
#   container is never exposed. Bring it back up by hand with:
#       rm -f /var/caddy/flags/maintenance.flag
#
# REQUIREMENTS
#   - lsof installed, and root privileges.
#   - FLAG below must point at the exact same file the Caddyfile @maintenance
#     matcher looks for.
#   - request_header X-Real-IP {remote_host} in that same Caddyfile block.
#
# USAGE
#   ./discourse-rebuild.sh
#
# TO SEE WHAT THE WATCHER CAUGHT
#   cat /var/log/unixip-hits.log
#

set -e

FLAG=/var/caddy/flags/maintenance.flag
SOCK=/var/discourse/shared/standalone/nginx.http.sock

cleanup() {
  local code=$?
  systemctl reload caddy
  if [ $code -eq 0 ]; then
    rm -f "$FLAG"
    echo "✅ Rebuild finished. Site is online."
  else
    echo "⚠️  Rebuild failed (exit code $code). The site is still in maintenance."
    echo "    Check it, and once it's ready: rm -f $FLAG"
  fi
}
trap cleanup EXIT

mkdir -p "$(dirname "$FLAG")"
touch "$FLAG"
echo "🔧 Maintenance is on. Waiting for in-flight requests..."

# Rough heuristic: count the processes holding the socket open and wait until
# only the listener is left. Up to 30s.
for i in $(seq 30); do
  n=$(lsof -t "$SOCK" 2>/dev/null | wc -l || echo 0)
  [ "$n" -le 1 ] && break
  sleep 1
done

/var/discourse/launcher rebuild app

# Up to 60 attempts: about 2 minutes of sleeps, more if any curl hits its own
# 5s timeout.
echo "⏳ Waiting for Discourse to answer..."
status=000
for i in $(seq 60); do
  status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
    --unix-socket "$SOCK" http://localhost/srv/status 2>/dev/null || echo 000)
  [ "$status" = "200" ] && break
  sleep 2
done

if [ "$status" != "200" ]; then
  echo "⚠️  Discourse never answered within the polling window (last status code: $status)."
  exit 1
fi

echo "✅ Discourse ready after ~$((i*2))s."

systemd-run --unit=unixip-watch --collect \
  /bin/bash -c "docker exec app tail -F /var/log/nginx/access.log | grep --line-buffered 'unix:' >> /var/log/unixip-hits.log"

What the script actually does

  1. Puts the site into maintenance through Caddy’s flag file.
  2. Waits for the socket to go quiet, then runs ./launcher rebuild app.
  3. Polls /srv/status over the socket until Unicorn answers 200. This is the critical part: it is what stops requests from reaching nginx while Unicorn is still booting.
  4. Reloads Caddy (belt and braces, it did not turn out to be critical).
  5. Clears the maintenance flag only if that poll succeeded.

Several weeks and several rebuilds later, I have not seen that error again.

PS: I really don’t know what the hell I did.