# Step-by-step Discourse migration with R2 and Cloudflare integration
**URL:** https://meta.discourse.org/t/step-by-step-discourse-migration-with-r2-and-cloudflare-integration/382491
**Category:** Sysadmins
**Tags:** migrations, s3, cloudflare
**Created:** [September 13, 2025, 1:31am UTC](https://meta.discourse.org/t/step-by-step-discourse-migration-with-r2-and-cloudflare-integration/382491 "2025-09-13T01:31:47Z")
**Posts on this page:** 1
**Page:** 1
### Author:  [@hel\_Sinki](https://meta.discourse.org/u/hel_Sinki)
#### Post date: [September 13, 2025, 1:31am UTC](https://meta.discourse.org/t/step-by-step-discourse-migration-with-r2-and-cloudflare-integration/382491/1 "2025-09-13T01:31:47Z")
This is a post-mortem/runbook of a real migration. I skip common Discourse prep (the official docs cover it). I focus on the exact switches, Cloudflare R2 gotchas, the rails/rake one-liners that mattered, what failed, and how to make the same move low-risk next time.
* * *
## Target end-state
- Discourse runs on the new host (Docker, single `app` container).
- Uploads + front-end assets live on Cloudflare R2:
- Bucket `discourse-uploads` (public)
- Bucket `discourse-backups` (private)
- R2 custom domain: `https://files.example.com` (created in **R2 → Custom domains** , not a manual cross-account CNAME).
* * *
## 0) DB backups that actually work (nightly **and** cutover)
Nightly backups are for **disaster recovery**. A last-minute backup is for **migration cutover**. Keep both.
### 0.1 Policy
- Nightly: **DB-only** backup (`.sql.gz`, no uploads) → **verify locally** → **upload to R2**. Keep ≥7 copies (or use R2 lifecycle).
- Cutover: right before DNS switch, make another **DB-only** backup and restore that to the new host to minimize content gap.
### 0.2 Make a DB-only backup and verify
Inside the container:
```bash
# Optional but nice: reduce writes while snapshotting
discourse enable_readonly
# Trigger a DB-only backup from Admin UI (uncheck "with uploads")
# or CLI:
discourse backup
# Verify the artifact
ls -lh /var/discourse/shared/standalone/backups/default/
zcat -t /var/discourse/shared/standalone/backups/default/.sql.gz
```
**Deep verify (best):** restore to a **temporary DB** and count rows:
```bash
cd /var/discourse && ./launcher enter app
sudo -E -u postgres psql -tc "DROP DATABASE IF EXISTS verifydb;"
sudo -E -u postgres createdb verifydb
zcat /shared/backups/default/.sql.gz | sudo -E -u postgres psql verifydb
sudo -E -u postgres psql -d verifydb -c "select count(*) from topics where deleted_at is null;"
sudo -E -u postgres psql -d verifydb -c "select count(*) from posts where post_type=1 and deleted_at is null;"
sudo -E -u postgres dropdb verifydb
exit
```
> If the gzip test or the temporary restore fails, **do not** upload that file to R2—fix and re-backup.
### 0.3 Push to R2 only after it passes
```bash
aws s3 cp /var/discourse/shared/standalone/backups/default/.sql.gz \
s3://discourse-backups/
```
### 0.4 Why sizes differ (1–4 GB is normal)
Both Admin nightly and manual `pg_dump` produce **DB-only** `.sql.gz`. Size differences usually come from included tables and compression, not “missing posts”. If you want to see what’s inside:
```bash
# Which tables have data in the dump?
zcat .sql.gz | grep -E '^COPY public\.' | awk '{print $2}' | sort -u | head
# Quick line-count approximation for key tables
zcat .sql.gz | awk '/^COPY public.posts /{c=1;next}/^\\\./{c=0} c' | wc -l
zcat .sql.gz | awk '/^COPY public.topics /{c=1;next}/^\\\./{c=0} c' | wc -l
```
> If those counts match expectations, the backup contains all posts/topics regardless of the file size.
* * *
## 1) Old host: prepare and copy the (verified) DB-only backup
Announce maintenance → enable read-only:
```bash
cd /var/discourse && ./launcher enter app
discourse enable_readonly
exit
```
Copy the verified `.sql.gz` to the new host:
```bash
rsync -avP -e "ssh -o StrictHostKeyChecking=no" \
root@OLD:/var/discourse/shared/standalone/backups/default/.sql.gz \
/var/discourse/shared/standalone/backups/default/
```
If you want an almost-zero content gap, repeat this step right before DNS cutover.
* * *
## 2) New host bootstrap
Install Docker + discourse\_docker:
```bash
apt-get update && apt-get install -y git curl tzdata
curl -fsSL https://get.docker.com | sh
systemctl enable --now docker
git clone https://github.com/discourse/discourse_docker /var/discourse
```
Create `containers/app.yml` with production values. Keep SSL templates commented until DNS points here. Minimum `env` set:
```yaml
env:
DISCOURSE_HOSTNAME: forum.example.com
# R2 / S3
DISCOURSE_USE_S3: "true"
DISCOURSE_S3_REGION: "auto"
DISCOURSE_S3_ENDPOINT: "https://.r2.cloudflarestorage.com"
DISCOURSE_S3_FORCE_PATH_STYLE: "true"
DISCOURSE_S3_BUCKET: "discourse-uploads"
DISCOURSE_S3_BACKUP_BUCKET: "discourse-backups"
DISCOURSE_S3_ACCESS_KEY_ID: ""
DISCOURSE_S3_SECRET_ACCESS_KEY: ""
DISCOURSE_S3_CDN_URL: "https://files.example.com"
DISCOURSE_BACKUP_LOCATION: "s3"
# R2 checksum knobs (prevent conflicts)
AWS_REQUEST_CHECKSUM_CALCULATION: "WHEN_REQUIRED"
AWS_RESPONSE_CHECKSUM_VALIDATION: "WHEN_REQUIRED"
# SMTP / Let’s Encrypt email
DISCOURSE_SMTP_ADDRESS: smtp.gmail.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: you@example.com
DISCOURSE_SMTP_PASSWORD: ""
DISCOURSE_SMTP_DOMAIN: example.com
DISCOURSE_NOTIFICATION_EMAIL: you@example.com
LETSENCRYPT_ACCOUNT_EMAIL: you@example.com
```
Publish assets to R2 during rebuild:
```yaml
hooks:
after_assets_precompile:
- exec:
cd: $home
cmd:
- sudo -E -u discourse bundle exec rake s3:upload_assets
- sudo -E -u discourse bundle exec rake s3:expire_missing_assets
```
Bring the container up (HTTP-only for now):
```bash
cd /var/discourse && ./launcher rebuild app
```
* * *
## 3) Restore the DB-only dump (`.sql.gz` via psql)
```bash
cd /var/discourse && ./launcher enter app
sv stop unicorn || true; sv stop sidekiq || true
# ensure a clean DB
sudo -E -u postgres psql -c "REVOKE CONNECT ON DATABASE discourse FROM public;"
sudo -E -u postgres psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='discourse';"
sudo -E -u postgres psql -c "DROP DATABASE IF EXISTS discourse;"
sudo -E -u postgres psql -c "CREATE DATABASE discourse WITH OWNER discourse TEMPLATE template0 ENCODING 'UTF8';"
sudo -E -u postgres psql -d discourse -c "CREATE EXTENSION IF NOT EXISTS citext;"
sudo -E -u postgres psql -d discourse -c "CREATE EXTENSION IF NOT EXISTS hstore;"
# import the dump
zcat /shared/backups/default/.sql.gz | sudo -E -u postgres psql discourse
sv start unicorn
[-d /etc/service/sidekiq] && sv start sidekiq || true
exit
```
If you’re still carrying local uploads pre-R2, you can rsync them once as a safety net; we’ll migrate them to R2 next.
* * *
## 4) R2 knobs that mattered
**Buckets & token** : create `discourse-uploads` (public) and `discourse-backups` (private). Bootstrap with an **Account API Token** scoped to those two buckets with **Admin Read & Write** (so `PutBucketCors` works), then rotate to **Object Read & Write** after success.
**Custom domain** : add `files.example.com` in **R2 → Custom domains** under the **same Cloudflare account** as your DNS zone (avoids 1014 cross-account CNAME errors).
**CORS** on `discourse-uploads`:
```json
[
{
"AllowedOrigins": ["https://forum.example.com","https://files.example.com"],
"AllowedMethods": ["GET","HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["*"],
"MaxAgeSeconds": 86400
}
]
```
Rebuild so CSS/JS/fonts publish to R2:
```bash
cd /var/discourse && ./launcher rebuild app
```
* * *
## 5) One-time migration of historical uploads to R2
```bash
cd /var/discourse && ./launcher enter app
yes "" | AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED \
sudo -E -u discourse RAILS_ENV=production bundle exec rake uploads:migrate_to_s3
```
If you get “X posts not remapped…”, see §7.2 for targeted fixes.
* * *
## 6) Switch production domain
Set in `app.yml`:
```yaml
DISCOURSE_HOSTNAME: forum.example.com
LETSENCRYPT_ACCOUNT_EMAIL: you@example.com
```
**DNS** : point `forum.example.com` to the new front (or origin) IP, enable SSL templates, then:
```bash
cd /var/discourse && ./launcher rebuild app
```
Sanity:
```bash
curl -I https://forum.example.com
./launcher logs app | tail -n 200
```
Seeing `HTTP/2 403` for anonymous usually means `login_required`—not an outage.
* * *
## 7) Things that actually broke (and fixes)
### 7.1 R2 checksum conflict
```plaintext
Aws::S3::Errors::InvalidRequest: You can only specify one non-default checksum at a time.
```
**Fix** (keep permanently):
```yaml
AWS_REQUEST_CHECKSUM_CALCULATION: "WHEN_REQUIRED"
AWS_RESPONSE_CHECKSUM_VALIDATION: "WHEN_REQUIRED"
```
### 7.2 “X posts are not remapped to new S3 upload URL”
**Reason** : some `cooked` HTML still points at `/uploads//original/...`.
**Targeted rebake:**
```bash
sudo -E -u discourse RAILS_ENV=production bundle exec rails r '
db = RailsMultisite::ConnectionManagement.current_db
ids = Post.where("cooked LIKE ?", "%/uploads/#{db}/original%").pluck(:id)
ids.each { |pid| Post.find(pid).rebake! }
puts "rebaked=#{ids.size}"
'
```
Or remap a static prefix then rebake touched posts:
```bash
sudo -E -u discourse RAILS_ENV=production bundle exec \
rake "posts:remap[/uploads/default/original,https://files.example.com/original]"
```
Re-run the migration to confirm clean:
```bash
yes "" | AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED \
sudo -E -u discourse RAILS_ENV=production bundle exec rake uploads:migrate_to_s3
```
### 7.3 Tasks “missing”
Always run with bundler + env:
```bash
sudo -E -u discourse RAILS_ENV=production bundle exec rake -T s3
sudo -E -u discourse RAILS_ENV=production bundle exec rake -T uploads
```
Print effective S3 settings:
```bash
sudo -E -u discourse RAILS_ENV=production bundle exec rails r \
'puts({ use_s3: ENV["DISCOURSE_USE_S3"], bucket: ENV["DISCOURSE_S3_BUCKET"], endpoint: ENV["DISCOURSE_S3_ENDPOINT"], cdn: ENV["DISCOURSE_S3_CDN_URL"] })'
```
### 7.4 `s3:upload_assets` AccessDenied
Use an **Admin RW** token for bootstrap (bucket-level CORS ops), then rotate to **Object RW**.
* * *
## 8) Verification
**Inside the container**
```bash
# URLs now using the CDN
sudo -E -u discourse RAILS_ENV=production bundle exec rails r \
'puts Upload.where("url LIKE ?", "%files.example.com%").limit(5).pluck(:url)'
# Remaining cooked references to local uploads (should trend to 0)
sudo -E -u discourse RAILS_ENV=production bundle exec rails r \
'db=RailsMultisite::ConnectionManagement.current_db; puts Post.where("cooked LIKE ?", "%/uploads/#{db}/original%").count'
```
**Browser**
- Network tab shows assets from `files.example.com`.
- Old topics show images under `https://files.example.com/original/...`.
**Backups**
- Admin → Backups → create one; confirm a new object appears in `discourse-backups` on R2.
* * *
## 9) Cleanup
When cooked references are essentially 0:
```bash
mv /var/discourse/shared/standalone/uploads /var/discourse/shared/standalone/uploads.bak
mkdir -p /var/discourse/shared/standalone/uploads
chown -R 1000:1000 /var/discourse/shared/standalone/uploads
# after a few stable days
rm -rf /var/discourse/shared/standalone/uploads.bak
```
Rotate secrets (R2 token → Object RW; SMTP app password if it ever hit logs).
* * *
## 10) Next time (playbook) — R2-first path
1. **Old → New (DB-only)**: read-only → backup → restore `.sql.gz` via `psql`.
2. **Wire R2 before DNS** : buckets, token (Admin RW → later Object RW), custom domain, CORS.
3. **`env` + `hooks`** : checksum flags + `s3:upload_assets`; rebuild.
4. **DNS cutover** to the new host.
5. **Migrate uploads** to R2.
6. **Fix stragglers** (targeted rebake/remap) → quick re-run of the migration.
7. **Sidekiq** finishes background rebakes (or `posts:rebake_uncooked_posts`).
8. **Backups** to R2 verified.
9. **Permissions hardening** and secret rotation.
10. **Cleanup** local uploads after a cooling-off period.
* * *
## Appendix A — “verify-before-upload” nightly (pseudo-cron)
```bash
LATEST=$(ls -1t /var/discourse/shared/standalone/backups/default/*.sql.gz | head -n1)
# 1) gzip integrity
gzip -t "$LATEST" || exit 1
# 2) temporary-DB row counts
cd /var/discourse && ./launcher enter app <<'EOS'
sudo -E -u postgres psql -tc "DROP DATABASE IF EXISTS verifydb;"
sudo -E -u postgres createdb verifydb
zcat /shared/backups/default/$(basename '"$LATEST"') | sudo -E -u postgres psql verifydb
sudo -E -u postgres psql -d verifydb -c "select count(*) as topics from topics where deleted_at is null;"
sudo -E -u postgres psql -d verifydb -c "select count(*) as posts from posts where post_type=1 and deleted_at is null;"
sudo -E -u postgres dropdb verifydb
exit
EOS
# 3) only then upload to R2
aws s3 cp "$LATEST" s3://discourse-backups/
```
* * *
## Appendix B — Minimal front proxy (optional)
A tiny reverse proxy VM in front can terminate TLS and forward to the origin over HTTPS. Replace IPs with your own.
**Upstream:** `/etc/nginx/conf.d/upstream.conf`
```nginx
upstream origin_forum {
server :443;
keepalive 64;
}
```
**Site:** `/etc/nginx/sites-available/forum.conf`
````nginx
server {
listen 80;
listen [::]:80;
server_name forum.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name forum.example.com;
ssl_certificate /etc/letsencrypt/live/forum.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/forum.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_timeout 1d;
client_max_body_size 100m;
add_header Strict-Transport-Security "max-age=31536000" always;
location / {
proxy_pass https://origin_forum;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host forum.example.com;
proxy_ssl_server_name on;
proxy_ssl_name forum.example.com;
# optional verification:
# proxy_ssl_verify on;
# proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_buffering off;
proxy_read_timeout 360s;
proxy_send_timeout 360s;
proxy_connect_timeout 60s;
add_header X-Relay relay-min always;
}
location /message-bus/ {
proxy_pass https://origin_forum;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host forum.example.com;
proxy_ssl_server_name on;
proxy_ssl_name forum.example.com;
proxy_buffering off;
proxy_read_timeout 3600s;
}
}
``
Enable & reload:
```bash
ln -sf /etc/nginx/sites-available/forum.conf /etc/nginx/sites-enabled/forum.conf
rm -f /etc/nginx/sites-enabled/default
nginx -t && systemctl reload nginx
````
**Quick check:**
```bash
curl -I https://forum.example.com # expect HTTP/2 200/302 and X-Relay header
```