복구 및 PSQL 업데이트를 위해 (임시) 네트워크 스토리지 사용

지원 자체 호스팅
1

저희는 VPS 환경에서 비교적 큰 규모의 Discourse를 운영 중이며, 현재까지는 잘 작동하고 있습니다. CPU/메모리 성능 측면에서는 여유가 있습니다. 다만 디스크 용량이 다소 부족합니다. 일상적인 운영에서는 문제가 되지 않지만, 예를 들어 Postgres를 업그레이드할 때(13에서 15로의 업그레이드가 용량 부족으로 아직 미완료 상태입니다) 용량이 부족하여 쉽게 확장할 수 없는 상황입니다.

Postgres 업데이트를 위한 다른 옵션들이 있다는 점은 알고 있지만, 이번 질문은 좀 더 일반적인 관점에서 드리는 것입니다.

저희는 Hetzner를 사용 중이며, 임시로 사용할 네트워크 스토리지가 쉽게 이용 가능합니다.

현재 테스트 서버에서 이를 임시로 구성해 보는 중입니다. 먼저 라이브 사이트의 백업을 복원하고, 이후 Postgres 업그레이드를 테스트할 예정입니다. 아직까지 성공하지는 못했습니다.

이미 심볼릭 링크를 시도해 보았지만 작동하지 않았으며, 여기 어딘가에서 이것이 권장되지 않는 방법이라는 것도 읽었습니다. 또한 /shared 공유 폴더를 /var/discourse/shared/standalone에서 /mnt/ext-storage/standalone으로 이동하고 파일을 옮겨 보았지만, 아쉽게도 문제가 발생했습니다. 빌드조차 완료할 수 없습니다.

이러한 경우를 위해 작동하는 방법이 있을까요? 해당 드라이브의 성능이 로컬 드라이브보다 훨씬 낮다는 점은 인지하고 있으며, 포럼을 그 위에 실행할 계획은 없습니다. 다만 특정 시나리오에서 이를 편하게 활용할 수 있는 방법을 찾고 싶습니다.

2

If your goal is to do the upgrade the easiest thing is to spin up a new vm and migrate to it. You skip the need to upgrade the database and you get a new os on your vm, which you likely need to do anyway.

Follow Move a Discourse site to another VPS with rsync and don’t copy the database (but yes uploads and let’s encrypt and certs.

If your backups are on s3 it’s really simple to freeze the old one, make a backup, and restore the backup on the new machine.

If hetzner has some kind of permanent ip that can be assigned to different servers, you don’t even need to change dns.

You want to know that you can build a new server so that if you ever have to for some reason you’ll be able to. This is a prefect chance to practice.

3

Actually this is not an option. It’s not running out of space for day to day needs anyways. Also we’re on a 600gb drive space right now and using ~50%. There is not a bigger option - at least not on Hetzner.

That’s why i explicitly was asking for the external drive.

4

Have you performed a ./launcher cleanup app out of interest? Has that not released enough space to perform the upgrade in place?

5

should that matter for a rebuild? For a simple restart I’d understand but for a rebuild?

6

I wasn’t suggesting to switch to a bigger drive, just get a new server exactly like that one. Install Discourse, restore your database.

That’s a very good question.

Yes. Every rebuild builds a new container and each of those takes up space. If you’ve never done it, you could have tens of GB freed.

7

For the database upgrade you need all the space you can get. So yes. I would say important.

8

Our update guide includes a guide for your exact use case

We added this option for people who in the same situation as you. Just be sure to store a backup offsite before trying this!

9

If you’re on a VM, it’s much, much easier to just move to a new one, and there are many benefits. Here are a few:

  • zero risk–if something goes wrong, you still have your old server
  • zero downtime (just read-only on the old server)
  • you get the OS upgrade that you probably need anyway
  • you can verify that you know how to spin up a new server should calamity strike
  • you don’t need to reindex and vacuum
10

Appreciate your advise guys.

Thanks, thats the other option that I said I’m aware of. Thanks anyways for pointing it out.

Not doubting that - still would have loved to explore the option of adding additional storage for maintenance tasks if needed.

11

It can be handy to keep your uploads, or, say /var/discourse/shared/web_only on network storage. You need to edit the yml file to point to it rather than use a symlink (the symlink doesn’t work because the container can’t access the place that your symlink points to.

Then if you do move to a new vm, you can just remount that network storage rather than copy it over.

I don’t recommend network storage for the database since it’s slower.

12

I think it’s worthwhile breaking down what your usage is. Your actual database size might not be so large, if most of your usage is uploads, and it’s only the database part which demands maybe 3x space during upgrade.

One thing you can check is the relative size of a backup with downloads compared to a backup without downloads.

Or, use the command line. Here’s some output from my rather small forum:

root@rc-debian-hel:~# free -m
               total        used        free      shared  buff/cache   available
Mem:            3813        1631         267         492        1915        1504
Swap:           4095         730        3365
root@rc-debian-hel:~# swapon 
NAME                       TYPE  SIZE   USED PRIO
/var/local/swap/swapfile.1 file 1024M 730.2M   -2
/var/local/swap/swapfile.3 file 1024M   136K   -3
/var/local/swap/swapfile.0 file 1024M     0B   -4
/var/local/swap/swapfile.2 file 1024M     0B   -5
root@rc-debian-hel:~# df -h 
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           382M  1.2M  381M   1% /run
/dev/sda1        38G   22G   14G  62% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      253M  6.3M  246M   3% /boot/efi
overlay          38G   22G   14G  62% /var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/merged
tmpfs           382M     0  382M   0% /run/user/0

Looking closer:

root@rc-debian-hel:~# du -kx / | sort -n | tail -33
767000	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share/pnpm
767004	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share
767020	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local
795804	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse
795808	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home
833836	/var/discourse/shared/standalone/postgres_data
884648	/var/discourse/shared/standalone/uploads/default/original
978000	/usr/lib/modules
991644	/var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/diff
991664	/var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766
1025164	/var/discourse/shared/standalone/uploads/default/optimized
1146528	/usr/lib/firmware
1350496	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff
1350512	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554
1909816	/var/discourse/shared/standalone/uploads/default
1919380	/var/discourse/shared/standalone/uploads
2471968	/usr/lib
2506940	/var/log/journal/82e4cf9bff9748d090b41e6d336353eb
2515140	/var/log/journal
2592200	/var/log
3029428	/usr
3839148	/var/discourse/shared/standalone/backups/default
3839152	/var/discourse/shared/standalone/backups
4194324	/var/local/swap
4194328	/var/local
5171844	/var/lib/docker/overlay2
5217052	/var/lib/docker
5455612	/var/lib
6682972	/var/discourse/shared/standalone
6682976	/var/discourse/shared
6685716	/var/discourse
19041368	/var
23037264	/
root@rc-debian-hel:~# du -kx /var/discourse/shared/ | sort -n | tail -33
42312	/var/discourse/shared/standalone/uploads/default/original/2X/f
42448	/var/discourse/shared/standalone/uploads/default/original/2X/1
42548	/var/discourse/shared/standalone/uploads/default/original/2X/6
43380	/var/discourse/shared/standalone/uploads/default/optimized/2X/2
44148	/var/discourse/shared/standalone/uploads/default/optimized/2X/5
44340	/var/discourse/shared/standalone/uploads/default/optimized/2X/1
45240	/var/discourse/shared/standalone/uploads/default/optimized/2X/e
46648	/var/discourse/shared/standalone/uploads/default/optimized/2X/c
49516	/var/discourse/shared/standalone/uploads/default/optimized/2X/8
49772	/var/discourse/shared/standalone/uploads/default/optimized/2X/9
49932	/var/discourse/shared/standalone/log/var-log/nginx
50788	/var/discourse/shared/standalone/uploads/default/optimized/2X/0
55428	/var/discourse/shared/standalone/uploads/default/optimized/2X/d
55844	/var/discourse/shared/standalone/uploads/default/optimized/2X/f
57548	/var/discourse/shared/standalone/uploads/default/optimized/2X/6
77280	/var/discourse/shared/standalone/log/var-log
81928	/var/discourse/shared/standalone/postgres_data/pg_wal
84064	/var/discourse/shared/standalone/log
294384	/var/discourse/shared/standalone/uploads/default/optimized/1X
325068	/var/discourse/shared/standalone/uploads/default/original/1X
559576	/var/discourse/shared/standalone/uploads/default/original/2X
724684	/var/discourse/shared/standalone/postgres_data/base/16384
730776	/var/discourse/shared/standalone/uploads/default/optimized/2X
749424	/var/discourse/shared/standalone/postgres_data/base
833836	/var/discourse/shared/standalone/postgres_data
884648	/var/discourse/shared/standalone/uploads/default/original
1025164	/var/discourse/shared/standalone/uploads/default/optimized
1909816	/var/discourse/shared/standalone/uploads/default
1919380	/var/discourse/shared/standalone/uploads
3839148	/var/discourse/shared/standalone/backups/default
3839152	/var/discourse/shared/standalone/backups
6682972	/var/discourse/shared/standalone
6682976	/var/discourse/shared/
13

I should have been more precise initially but didn’t expect to get such widespread feedback.

Our uploads and backups are on S3. The database size on the live system as at around 230GB now. Compressed backups are around 25GB I think.

14

Wow that is one of the big ones! At that size the database operation is usually a bit more troublesome indeed.

15

Yow. Have you done a vacuum lately?

16

Nope, I was under the impression that autovacuum should take care about it? No?

17

I think it’s supposed to. I am not clear what triggers it. I think doing an extra one doesn’t hurt and might save you some space. It’s recommended after you do the upgrade if you do it in place. I’ve seen it clean up considerable space a few times.

If your database is 230GB, I’d definitely restore it to a new server. Downtime reading and writing 230GB is going to be significant.