Do your rebuilds take forever with millions of uploads? Try adding this template!

Support Self-hosting
1

parallel-fs-ops.template.yml (2.8 KB)

Let’s talk about a scaling problem that becomes painfully obvious once a Discourse site accumulates a large upload library.

What used to take minutes for a chown command to run across a huge uploads directory now takes seconds!

Background

Discourse rebuilds can execute recursive operations such as:

chown -R ...
chmod -R ...

More specifically, this line in templates/web.template.yml:

- chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp

These commands walk the filesystem serially, one-by-one.

That is perfectly reasonable for a small installation. But when shared/uploads contains hundreds of thousands—or millions—of files, recursive ownership and permission operations can dominate a deploy. CPU, storage, and network capacity may be available, yet one process walks the entire tree one inode at a time.

For upload-heavy communities, the result can be:

  • Extremely long rebuilds
  • Longer maintenance windows
  • Delayed deployments and security updates
  • Poor utilization of fast or distributed storage
  • A deploy appearing stuck while it processes an enormous file tree
  • Especially painful performance on NFS, JuiceFS, CephFS, and other remote filesystems

The frustrating part is that many of these files are independent. Their permissions can be processed concurrently.

The Solution: Parallel Filesystem Operations Template

I created a pups template that transparently replaces recursive chmod and chown operations with parallel find and xargs pipelines.

The wrappers announce themselves whenever they intercept a recursive operation:

echo "[parallel-fs-ops] chmod -R override active: $*" >&2

and:

echo "[parallel-fs-ops] chown -R override active: $*" >&2

That bracketed prefix makes the optimization easy to spot in a deploy log.

What a deploy looks like

Near the beginning of the deploy, the template confirms which binaries will handle later filesystem operations:

[parallel-fs-ops] chmod -> /usr/local/bin/chmod
[parallel-fs-ops] chown -> /usr/local/bin/chown

When an upstream template later runs a recursive permission change, the deploy output includes a line similar to:

[parallel-fs-ops] chmod -R override active: -R 0755 /var/www/discourse/public

A recursive ownership change produces:

[parallel-fs-ops] chown -R override active: -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp

For an upload-heavy installation, you might see something resembling:

[parallel-fs-ops] chown -R override active: -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp

The exact paths and arguments depend on the templates being used, but the important part is the visible marker:

[parallel-fs-ops]

Without the template, the deploy may appear to pause for a long time during a recursive filesystem operation. With the template, the log tells you that:

  1. The wrapper was installed correctly.
  2. A recursive operation was detected.
  3. The parallel implementation is active.
  4. The original arguments being processed are visible.

This is particularly valuable during troubleshooting because it distinguishes a slow parallel filesystem traversal from a hung build.

After the operation finishes, the deployment continues with its normal pups output. The wrapper itself does not print one line per file, so even a tree containing millions of uploads does not flood the deploy log.

The template

run:
  - file:
      path: /usr/local/bin/chmod
      chmod: "+x"
      contents: |
        #!/bin/bash
        if [[ "$*" =~ (^|[[:space:]])-R([[:space:]]|$) ]]; then
          echo "[parallel-fs-ops] chmod -R override active: $*" >&2
          args=()
          for arg in "$@"; do
            [[ "$arg" != "-R" ]] && args+=("$arg")
          done
          mode="${args[0]}"
          targets=("${args[@]:1}")
          [[ ${#targets[@]} -eq 0 ]] && targets=(".")
          find "${targets[@]}" -print0 |
            xargs -0 -n 32 -P 128 /bin/chmod "$mode"
        else
          exec /bin/chmod "$@"
        fi

  - file:
      path: /usr/local/bin/chown
      chmod: "+x"
      contents: |
        #!/bin/bash
        if [[ "$*" =~ (^|[[:space:]])-R([[:space:]]|$) ]]; then
          echo "[parallel-fs-ops] chown -R override active: $*" >&2
          args=()
          for arg in "$@"; do
            [[ "$arg" != "-R" ]] && args+=("$arg")
          done
          owner="${args[0]}"
          targets=("${args[@]:1}")
          [[ ${#targets[@]} -eq 0 ]] && targets=(".")
          find "${targets[@]}" -print0 |
            xargs -0 -n 32 -P 128 /bin/chown "$owner"
        else
          exec /bin/chown "$@"
        fi

  - exec:
      cmd: |
        echo "[parallel-fs-ops] chmod -> $(command -v chmod)"
        echo "[parallel-fs-ops] chown -> $(command -v chown)"

The template installs wrappers in /usr/local/bin, which normally appears before /bin in PATH.

When a normal, non-recursive operation is requested, the wrapper delegates directly to the standard utility:

exec /bin/chmod "$@"

When -R is present, it removes the recursive flag, enumerates the targets safely with null delimiters, and processes batches concurrently:

find "${targets[@]}" -print0 |
  xargs -0 -n 32 -P 128 /bin/chmod "$mode"

This also works when pups invokes commands through /bin/sh. The wrapper’s Bash shebang is honored when the executable is launched, even though the calling shell is Dash.

Why this matters most when you have many uploads

Upload-heavy communities are exactly where deploy behavior needs to scale gracefully.

A long-running forum may contain:

  • Images embedded across years of posts
  • Avatars and profile backgrounds
  • Original and optimized image variants
  • Video and audio attachments
  • Documents and archives
  • Secure uploads
  • Plugin-managed media
  • Multisite upload trees

The amount of application code may remain relatively stable while the number of uploaded filesystem objects continues to grow. Filesystem traversal—not compilation or container creation—can eventually become the dominant deploy cost.

This is an unusual scaling problem: the more successful and content-rich the community becomes, the more expensive routine operational work can become.

Why a template is needed

Changing .bashrc or setting BASH_ENV does not reliably solve this. pups executes run commands through /bin/sh, and Dash neither loads Bash configuration nor understands Bash-specific functions.

A template provides a repeatable way to install the wrappers early enough that subsequent recursive operations—including those from upstream templates—resolve through the parallel implementation:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "containers/parallel-fs-ops.template.yml"

Configurable Options

Parallel processing options

The template uses:

find "${targets[@]}" -print0 |
  xargs -0 -n 32 -P 128 /bin/chmod "$mode"

The relevant xargs parameters are:

Option Purpose
-0 Reads null-delimited paths produced by find -print0. This safely handles filenames containing spaces, quotes, tabs, or newlines.
-n 32 Passes at most 32 paths to each chmod or chown invocation. This is the batch size.
-P 128 Allows up to 128 chmod or chown processes to run concurrently. This is the parallelism level.

Together, -n 32 -P 128 means that as many as 128 processes can run simultaneously, with each process handling a batch of up to 32 paths. Roughly 4,096 paths may therefore be actively distributed across command batches at once.

Choosing -n

-n controls how much work is assigned to each command:

  • Lower values provide finer work distribution but start more processes.
  • Higher values reduce process-launch overhead but create larger, less evenly distributed batches.
  • -n 1 runs one chmod or chown command per path.
  • -n 32 is a reasonable starting point for balancing batching and parallelism.
  • Very large values may reduce the effectiveness of -P because fewer total batches are created.

Choosing -P

-P controls how many commands may run at the same time:

  • Lower values reduce load on the CPU and filesystem.
  • Higher values can improve performance on fast or distributed storage.
  • Excessive parallelism can overwhelm disks, saturate a metadata server, or make performance worse.
  • -P 1 is effectively serial execution.
  • -P 8 or -P 16 is a conservative starting point.
  • -P 32 may suit fast SSD-backed storage.
  • -P 128 should be used only when the filesystem and host can sustain that concurrency.

The best values depend on filesystem latency, metadata performance, CPU capacity, and the number of files. Both should ideally be configurable and benchmarked for the specific installation.

Too much parallelism can overwhelm a filesystem, saturate metadata servers, or degrade deployment performance. Batch size and concurrency should therefore be configurable.

This template is a practical workaround, but the larger proposal is broader:

Could Discourse officially support configurable parallelism for large recursive filesystem operations during deploys?

An upstream implementation could:

  • Parallelize only known large directory trees
  • Avoid traversing unchanged upload trees unnecessarily
  • Make concurrency configurable
  • Detect local versus network-backed filesystems
  • Preserve complete chmod and chown argument semantics
  • Emit periodic progress for very large trees
  • Record timings so administrators can identify deployment bottlenecks

Important caution

The wrapper above is focused on the recursive command forms used by our build process. It is not a complete reimplementation of every possible chmod or chown option combination.

It should be tested against the exact commands generated by a site’s templates before production use. Operators should start with conservative parallelism and measure the effect on their storage.

But the underlying problem is real: serial recursive metadata operations do not scale well when a community has accumulated a massive upload tree.

Good Luck, and I appreciate any comments or suggestions (even if maybe I duplicated someone else’s efforts, I’d appreciate pointers to that as well)!

Cheers!

2 Likes