Michael,
Now that I have used it, I agree that restic is “the best” for backups, but I’m curious if given recent minio controversy have you used a compatible option like garage in its place? Or is the answer “it doesn’t matter, they all use the same commands, so you can use whichever you like”? Those are tools that I’m only just now going to learn, so I don’t know the detailed differences, only that current guidance is to use garage over minio.
Oh, yeah, I am confident that I’d start with garage today. I just haven’t migrated yet.
Also thanks to @raykholo for pointing out to me out of band that my instructions for disabling transparent huge pages were quietly wrong.
Anyone tracking this, please use this instead:
echo 'w /sys/kernel/mm/transparent_hugepage/enabled - - - - never
w /sys/kernel/mm/transparent_hugepage/defrag - - - - never' > /etc/tmpfiles.d/thp.conf
systemd-tmpfiles --create
These are not configurable via sysctl, and if you have previously created /etc/sysctl.d/10-huge-pages.conf following my earlier instructions, you can delete it.
I didn’t validate that correctly when I wrote it, and I’ve failed to notice ever since that it was not working correctly. Nice catch, and thank you!
Hmm. I have two systems, and I’ve just looked at them, and I might hypothesise that for Ubuntu 22 you’re right, and your correction has helped, but for Ubuntu 24 it isn’t needed. In writing this, I realise that your correction might also be effective in Ubuntu 24 even if it isn’t needed. What version of OS are you working with?
It’s a question of the default configuration. It’s safe to disable if it’s already disabled (no-op) and the method now described is widely supported.
I’m running AlmaLinux, as described at the top. Once docker finally got support for v2 namespaces, I took my first opportunity to move to a Red Hat derivative OS.
Perhaps answering my own question, here are some notes from trying to use Garage with your runbook:
1. Garage doesn’t support “permission tags” on files (this is the real Garage gap)
Amazon lets you tag each file “public” or “private” individually. Garage doesn’t implement that.
Discourse tries to use it by default. The sneaky part: when Discourse said “save this file, mark it public,” Garage accepted the request and quietly ignored the tag. So basic uploads looked fine, and it would only have broken much later, the first time someone made an upload private.
Fix: tell Discourse to stop using tags (one setting). Cloudflare’s R2 has the same gap and gets the same fix. Garage handles permissions its own way, at the bucket level, which is all we need.
2. Discourse insists on naming buckets a specific way (not a Garage flaw)
Discourse won’t say “the server at garage, bucket uploads.” It insists on uploads.garage — bucket name glued to the front, like a subdomain. And there’s no option to turn that off.
Nothing on our network knew that name, so Discourse couldn’t connect at all — the install died partway through.
Fix: gave Garage that naming style and registered the names. Two lines of config.
This one is a known Discourse quirk, not a Garage one — it’s why Oracle’s storage is on Discourse’s official “won’t work” list.
3. Images needed a public web address (nothing to do with Garage)
Discourse writes each image’s address into the post permanently. Without telling it the public address up front, it saved internal addresses only our server can reach — so every image would be broken for visitors, forever, unless we rebuilt every post.
Fix: set the CDN address before the first upload. Would’ve been identical on Amazon, R2, anything.
Note: every one of these was silent. Nothing said “unsupported.” One accepted the request and ignored it, one looked like a network error, one looked like it worked fine.
So, just some initial things to keep in mind for a new user trying this for the first time, and/ or someone trying to follow Michael’s runbook with Garage.
Here are the full AI notes from what we experienced now that the forum has been deployed:
Running Discourse on self-hosted Garage (S3) behind a Cloudflare tunnel — notes
Garage isn’t on the “Configure an S3 compatible object storage provider” compatibility table, so here’s a data point. Setup: Garage v2.3.0, two-container Discourse, Debian 13,
single node, Cloudflare tunnel instead of an external nginx. Caveat up front: this is one
small deployment with no production traffic yet, so treat it as “works” and not “battle-
tested at Maker Forums scale”.
It works. Verified against Discourse’s own code paths, not a CLI: UploadCreator, OptimizedImage, ListObjectsV2, HeadObject/GetObject with byte-exact round trip, multipart, delete, remove_upload, plus BackupRestore::Backuper writing to the backups bucket and BackupStore#files / #download_file reading it back. Lifecycle configuration works, so s3_configure_tombstone_policy actually takes effect rather than being silently ignored. PutBucketCors works, so manual CORS is fine.
You must configure virtual-host-style addressing, and it will break db:migrate if you don’t. Discourse turns endpoint http://garage:3900 + bucket “uploads” into the host uploads.garage:3900, and there’s no path-style option anywhere in s3_helper.rb or site_settings.yml. Two halves are required: root_domain under [s3_api] in garage.toml, AND a resolvable DNS name per bucket (a Docker network alias per bucket, since Docker DNS has no wildcards). Symptom if you miss it is an Aws::Waiters error about getaddrinfo during SiteIconManager.ensure_optimized! — reads like a network fault, not a storage one. Every new bucket needs a new name.
Garage does not implement PutObjectAcl / GetObjectAcl, so set s3_use_acls to false — same as R2. Two traps here. First, PutObject with --acl public-read is silently ACCEPTED and the header ignored, so basic upload tests pass and it only breaks later on a secure-upload transition. Second, DISCOURSE_S3_USE_ACLS is not a shadowed global — putting it in app.yml does nothing. It came up true on my first boot despite being in the env. It has to be a site setting, and it needs re-checking after every rebuild because nothing warns you.
If you put a CDN in front, point it at Garage’s WEB endpoint (:3902), not the S3 API (:3900). Anonymous reads only work on the web endpoint; the S3 API correctly 403s unauthenticated requests, so a CDN aimed at :3900 fails on every image while your credentials are perfectly valid. You also need
garage bucket website --allowon the uploads bucket, and a root_domain under [s3_web]. Enable it on uploads only — the backups bucket must never be anonymously readable.ListObjectVersions returns NotImplemented on Garage. It doesn’t matter: grepping s3_helper.rb and file_store/s3_store.rb for list_object_versions / object_versions returns zero hits. Discourse doesn’t use S3 object versioning; tombstone expiry is lifecycle rules plus a prefix.
Testing with aws-cli or mc does NOT prove compatibility. Both default to path-style addressing against a custom endpoint; Discourse only does virtual-host style. My CLI pass came back 15/16 and looked like a green light, then the real install failed immediately on the addressing difference. Same store, same credentials, same operations. If you’re evaluating an unproven S3 backend, drive real Discourse — a throwaway single-container instance is enough, and doing it before any content exists is the whole point, since S3 → local is a one-way door.
On a Cloudflare tunnel specifically: the external-nginx-for-SSL section doesn’t apply (TLS terminates at the edge, no certbot, no inbound 80/443), but the real-IP outlet config is still essential and arguably more so. Use real_ip_header CF-Connecting-IP rather than X Forwarded-For — when the tunnel is the only ingress path it’s a single unambiguous edge-set value. Verify it by posting from a known public IP and checking what nginx logged; without it Discourse records the connector’s Docker address for every request and rate-limits the entire internet as one client. What you lose versus the external-nginx approach is the maintenance page during rebuilds — cloudflared can’t serve one.
Minor correction to something widely (mis)stated, including by me: DISCOURSE_S3_CDN_URL is not baked irreversibly into stored URLs. upload.url does store the raw S3 URL with the internal host, but Discourse substitutes the CDN host at render time via Discourse.store.cdn_url / UrlHelper.cook_url. Setting it late is recoverable — the cost is
rake posts:rebake, because posts.cooked caches the HTML from cook time. Still set it before the first upload; just don’t panic if you didn’t.Two operational notes unrelated to Garage. ./launcher echoes the full docker run line including DISCOURSE_S3_SECRET_ACCESS_KEY in plaintext, so bootstrap logs are secret-bearing — worth rotating keys after a noisy install. And a single-node Garage still requires
layout assign+layout applybefore it will store anything, with replication_factor = 1 meaning no replication at all, so your backup job is the only durability you have.
I do not use an object store for public access in Discourse, due in part to the migration being a one-way door and also not needing it; in my case, it would be serving that data from the same storage, just with another VM to manage. I use an object store only as a restic backup target, done outside Discourse.
Discourse backups with thumbnails but without images, combined with host-level backups of uploads, backed up with restic on host, means that those limitations are, as far as I know, irrelevant.