parallel-fs-ops.template.yml (2.8 KB) Let’s talk about a scaling problem that becomes painfully obvious once a Discourse site accumulates a large upload library. What used to take minutes for a chown command to run across a huge uploads directory now takes seconds! Background Discourse rebuilds can execute recursive operations such as: chown -R ... chmod -R ... More specifically, this line in templates/web.template.yml: - chown -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp These commands walk the filesystem serially, one-by-one. That is perfectly reasonable for a small installation. But when shared/uploads contains hundreds of thousands—or millions—of files, recursive ownership and permission operations can dominate a deploy. CPU, storage, and network capacity may be available, yet one process walks the entire tree one inode at a time. For upload-heavy communities, the result can be: Extremely long rebuilds Longer maintenance windows Delayed deployments and security updates Poor utilization of fast or distributed storage A deploy appearing stuck while it processes an enormous file tree Especially painful performance on NFS, JuiceFS, CephFS, and other remote filesystems The frustrating part is that many of these files are independent. Their permissions can be processed concurrently. The Solution: Parallel Filesystem Operations Template I created a pups template that transparently replaces recursive chmod and chown operations with parallel find and xargs pipelines. The wrappers announce themselves whenever they intercept a recursive operation: echo "[parallel-fs-ops] chmod -R override active: $*" >&2 and: echo "[parallel-fs-ops] chown -R override active: $*" >&2 That bracketed prefix makes the optimization easy to spot in a deploy log. What a deploy looks like Near the beginning of the deploy, the template confirms which binaries will handle later filesystem operations: [parallel-fs-ops] chmod -> /usr/local/bin/chmod [parallel-fs-ops] chown -> /usr/local/bin/chown When an upstream template later runs a recursive permission change, the deploy output includes a line similar to: [parallel-fs-ops] chmod -R override active: -R 0755 /var/www/discourse/public A recursive ownership change produces: [parallel-fs-ops] chown -R override active: -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp For an upload-heavy installation, you might see something resembling: [parallel-fs-ops] chown -R override active: -R discourse:www-data /shared/log/rails /shared/uploads /shared/backups /shared/tmp The exact paths and arguments depend on the templates being used, but the important part is the visible marker: [parallel-fs-ops] Without the template, the deploy may appear to pause for a long time during a recursive filesystem operation. With the template, the log tells you that: The wrapper was installed correctly. A recursive operation was detected. The parallel implementation is active. The original arguments being processed are visible. This is particularly valuable during troubleshooting because it distinguishes a slow parallel filesystem traversal from a hung build. After the operation finishes, the deployment continues with its normal pups output. The wrapper itself does not print one line per file, so even a tree containing millions of uploads does not flood the deploy log. The template run: - file: path: /usr/local/bin/chmod chmod: "+x" contents: | #!/bin/bash if [[ "$*" =~ (^|[[:space:]])-R([[:space:]]|$) ]]; then echo "[parallel-fs-ops] chmod -R override active: $*" >&2 args=() for arg in "$@"; do [[ "$arg" != "-R" ]] && args+=("$arg") done mode="${args[0]}" targets=("${args[@]:1}") [[ ${#targets[@]} -eq 0 ]] && targets=(".") find "${targets[@]}" -print0 | xargs -0 -n 32 -P 128 /bin/chmod "$mode" else exec /bin/chmod "$@" fi - file: path: /usr/local/bin/chown chmod: "+x" contents: | #!/bin/bash if [[ "$*" =~ (^|[[:space:]])-R([[:space:]]|$) ]]; then echo "[parallel-fs-ops] chown -R override active: $*" >&2 args=() for arg in "$@"; do [[ "$arg" != "-R" ]] && args+=("$arg") done owner="${args[0]}" targets=("${args[@]:1}") [[ ${#targets[@]} -eq 0 ]] && targets=(".") find "${targets[@]}" -print0 | xargs -0 -n 32 -P 128 /bin/chown "$owner" else exec /bin/chown "$@" fi - exec: cmd: | echo "[parallel-fs-ops] chmod -> $(command -v chmod)" echo "[parallel-fs-ops] chown -> $(command -v chown)" The template installs wrappers in /usr/local/bin, which normally appears before /bin in PATH. When a normal, non-recursive operation is requested, the wrapper delegates directly to the standard utility: exec /bin/chmod "$@" When -R is present, it removes the recursive flag, enumerates the targets safely with null delimiters, and processes batches concurrently: find "${targets[@]}" -print0 | xargs -0 -n 32 -P 128 /bin/chmod "$mode" This also works when pups invokes commands through /bin/sh. The wrapper’s Bash shebang is honored when the executable is launched, even though the calling shell is Dash. Why this matters most when you have many uploads Upload-heavy communities are exactly where deploy behavior needs to scale gracefully. A long-running forum may contain: Images embedded across years of posts Avatars and profile backgrounds Original and optimized image variants Video and audio attachments Documents and archives Secure uploads Plugin-managed media Multisite upload trees The amount of application code may remain relatively stable while the number of uploaded filesystem objects continues to grow. Filesystem traversal—not compilation or container creation—can eventually become the dominant deploy cost. This is an unusual scaling problem: the more successful and content-rich the community becomes, the more expensive routine operational work can become. Why a template is needed Changing .bashrc or setting BASH_ENV does not reliably solve this. pups executes run commands through /bin/sh, and Dash neither loads Bash configuration nor understands Bash-specific functions. A template provides a repeatable way to install the wrappers early enough that subsequent recursive operations—including those from upstream templates—resolve through the parallel implementation: templates: - "templates/postgres.template.yml" - "templates/redis.template.yml" - "templates/web.template.yml" - "containers/parallel-fs-ops.template.yml" Configurable Options Parallel processing options The template uses: find "${targets[@]}" -print0 | xargs -0 -n 32 -P 128 /bin/chmod "$mode" The relevant xargs parameters are: Option Purpose -0 Reads null-delimited paths produced by find -print0. This safely handles filenames containing spaces, quotes, tabs, or newlines. -n 32 Passes at most 32 paths to each chmod or chown invocation. This is the batch size. -P 128 Allows up to 128 chmod or chown processes to run concurrently. This is the parallelism level. Together, -n 32 -P 128 means that as many as 128 processes can run simultaneously, with each process handling a batch of up to 32 paths. Roughly 4,096 paths may therefore be actively distributed across command batches at once. Choosing -n -n controls how much work is assigned to each command: Lower values provide finer work distribution but start more processes. Higher values reduce process-launch overhead but create larger, less evenly distributed batches. -n 1 runs one chmod or chown command per path. -n 32 is a reasonable starting point for balancing batching and parallelism. Very large values may reduce the effectiveness of -P because fewer total batches are created. Choosing -P -P controls how many commands may run at the same time: Lower values reduce load on the CPU and filesystem. Higher values can improve performance on fast or distributed storage. Excessive parallelism can overwhelm disks, saturate a metadata server, or make performance worse. -P 1 is effectively serial execution. -P 8 or -P 16 is a conservative starting point. -P 32 may suit fast SSD-backed storage. -P 128 should be used only when the filesystem and host can sustain that concurrency. The best values depend on filesystem latency, metadata performance, CPU capacity, and the number of files. Both should ideally be configurable and benchmarked for the specific installation. Too much parallelism can overwhelm a filesystem, saturate metadata servers, or degrade deployment performance. Batch size and concurrency should therefore be configurable. This template is a practical workaround, but the larger proposal is broader: Could Discourse officially support configurable parallelism for large recursive filesystem operations during deploys? An upstream implementation could: Parallelize only known large directory trees Avoid traversing unchanged upload trees unnecessarily Make concurrency configurable Detect local versus network-backed filesystems Preserve complete chmod and chown argument semantics Emit periodic progress for very large trees Record timings so administrators can identify deployment bottlenecks Important caution The wrapper above is focused on the recursive command forms used by our build process. It is not a complete reimplementation of every possible chmod or chown option combination. It should be tested against the exact commands generated by a site’s templates before production use. Operators should start with conservative parallelism and measure the effect on their storage. But the underlying problem is real: serial recursive metadata operations do not scale well when a community has accumulated a massive upload tree. Good Luck, and I appreciate any comments or suggestions (even if maybe I duplicated someone else’s efforts, I’d appreciate pointers to that as well)! Cheers!

Yup. But not everyone wants to add that complexity. I kind of am using S3/R2/B2 via using JuiceFS. That way it mounts locally like a real file system, and no worries. And then Cloudflare caches the assets that way automagicallly.

Should this instead be a PR that changes the templates doing the chown?

Yes please! I’ve moved this out of #wiki and back to #self-hosting, since it’s not really a pattern we want to encourage everyone to handle manually. If there’s a safe performance improvement we can make in the upstream template, let’s do that! I don’t think we’d want to override the system-level chown/chmod executables. But perhaps the appropriate find commands could be used directly?

Let’s try a simpler version first. This doesn’t run anything in parallel, but it does skip the chown for files which already have the correct permissions. It’s a pattern that we’re already using elsewhere in discourse_docker: PERF: Skip chown for files that already have the right ownership (#1115) main ← avoid-redundant-chown merged 01:23PM - 19 Aug 26 UTC (UTC) [image] davidtaylorhq +1 -1 The web template runs `chown -R discourse:www-data` over many directories on eve…ry rebuild. This rewrites the ownership even when it's already correct. Instead, we can use `find` to select only the files with wrong ownership. We already use this pattern for `$home` in the `web:` hook of this template. (see aef8682f1e9961d82b98aab87c7e47e0f08a6c52)

Just saw this and saw the PR. Definitely a nice compromise but huge perf gain for sure. It’s always the little things… :wink:

Rebuilds taking a long time with a large number of uploads (chown performance)

Product Support Self-hosting

NateDhaliwal (Nate Dhaliwal) August 17, 2026, 11:02pm 2

Nice. I guess that’s why S3 and CDN are things.

Topic		Replies	Views
Make rebuild with many uploads more efficient? Product Support	3	617	March 13, 2020
Stuck @ CHOWN for /var/www/discourse Self-hosting	15	3102	May 31, 2018
Skipping an step in discourse rebuild Product Support	1	516	September 19, 2020
Slow Discourse Rebuild on a VM with slow disks Self-hosting	7	1284	February 12, 2021
Этот файл слишком большой Self-hosting	29	966	May 4, 2026

Rebuilds taking a long time with a large number of uploads (chown performance)

Related topics