helmi
(Helmi)
February 12, 2025, 12:30pm
1
We’re running a bigger Discourse installation on a VPS setup that basically works very well for us. In terms of CPU/Mem performance we have headroom. Disk space though is a bit of an issue - not in the day to day business but when it comes to upgrading Postgres for example (the 13->15 Upgrade is outstanding because of that) we lack the space and can’t easily expand.
I know there are other options for the postgres update but consider this more of a general question.
We’re running on Hetzner where Network storage is easily available for temporary use.
On our test server i’m now playing around with making it work in a temporary way - first to restore a backup from the live site, later to test the Postgres Upgrade. So far I haven’t been successful.
I already tried symlinking but noticed it didn’t work and also read here somewhere that it’s not the recommended way. I have also tried moving the /shared share from /var/discourse/shared/standalone to /mnt/ext-storage/standalone and moved the files over there - unfortunately not without issues. I can’t even finish the build.
Is there any way that works for these kind of cases? I’m aware that the drive performance is much worse than the local drive but I’m not planning to run the forum on it. I’d really just like to have a comfortable way to use it for certain scenarios.
pfaffman
(Jay Pfaffman)
February 12, 2025, 2:31pm
2
If your goal is to do the upgrade the easiest thing is to spin up a new vm and migrate to it. You skip the need to upgrade the database and you get a new os on your vm, which you likely need to do anyway.
Follow Move a Discourse site to another VPS with rsync and don’t copy the database (but yes uploads and let’s encrypt and certs.
If your backups are on s3 it’s really simple to freeze the old one, make a backup, and restore the backup on the new machine.
If hetzner has some kind of permanent ip that can be assigned to different servers, you don’t even need to change dns.
You want to know that you can build a new server so that if you ever have to for some reason you’ll be able to. This is a prefect chance to practice.
helmi
(Helmi)
February 12, 2025, 2:41pm
3
Actually this is not an option. It’s not running out of space for day to day needs anyways. Also we’re on a 600gb drive space right now and using ~50%. There is not a bigger option - at least not on Hetzner.
That’s why i explicitly was asking for the external drive.
merefield
(Robert)
February 12, 2025, 2:59pm
4
Have you performed a ./launcher cleanup app
out of interest? Has that not released enough space to perform the upgrade in place?
1 Like
helmi
(Helmi)
February 12, 2025, 3:00pm
5
should that matter for a rebuild? For a simple restart I’d understand but for a rebuild?
pfaffman
(Jay Pfaffman)
February 12, 2025, 3:01pm
6
I wasn’t suggesting to switch to a bigger drive, just get a new server exactly like that one. Install Discourse, restore your database.
That’s a very good question.
Yes. Every rebuild builds a new container and each of those takes up space. If you’ve never done it, you could have tens of GB freed.
1 Like
merefield
(Robert)
February 12, 2025, 3:01pm
7
For the database upgrade you need all the space you can get. So yes. I would say important.
Falco
(Falco)
February 12, 2025, 3:49pm
8
Our update guide includes a guide for your exact use case
Doing a manual update / space constrained environments
If you are in a constrained space environment without any way to get more space you can try the following:
./launcher stop app #(or both web_only and data if that is your case)
mkdir -p /var/discourse/shared/standalone/postgres_data_new
docker run --rm \
--entrypoint=/bin/bash \
-v /var/discourse/shared/standalone/postgres_data:/var/lib/postgresql/13/data \
-v /var/discourse/shared/standalone/postgres_data_new:/var/lib/postgresql/15/data \
tianon/postgres-upgrade:13-to-15 \
-c "apt-get update && apt-get install -y postgresql-13-pgvector postgresql-15-pgvector &&
docker-upgrade"
mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
mv /var/discourse/shared/standalone/postgres_data_new /var/discourse/shared/standalone/postgres_data
docker run --rm -v /var/discourse/shared/standalone:/shared \
local_discourse/app chown -R postgres:postgres /shared/postgres_data #(or local_discourse/data)
./launcher rebuild app #(or first data and then web_only if that is your case)
On my tests this procedure requires less than 1x your current database size in free space.
We added this option for people who in the same situation as you. Just be sure to store a backup offsite before trying this!
2 Likes
pfaffman
(Jay Pfaffman)
February 12, 2025, 6:11pm
9
If you’re on a VM, it’s much, much easier to just move to a new one, and there are many benefits. Here are a few:
zero risk–if something goes wrong, you still have your old server
zero downtime (just read-only on the old server)
you get the OS upgrade that you probably need anyway
you can verify that you know how to spin up a new server should calamity strike
you don’t need to reindex and vacuum
1 Like
helmi
(Helmi)
February 13, 2025, 8:29am
10
Appreciate your advise guys.
Thanks, thats the other option that I said I’m aware of. Thanks anyways for pointing it out.
Not doubting that - still would have loved to explore the option of adding additional storage for maintenance tasks if needed.
2 Likes
pfaffman
(Jay Pfaffman)
February 13, 2025, 10:20am
11
It can be handy to keep your uploads, or, say /var/discourse/shared/web_only on network storage. You need to edit the yml file to point to it rather than use a symlink (the symlink doesn’t work because the container can’t access the place that your symlink points to.
Then if you do move to a new vm, you can just remount that network storage rather than copy it over.
I don’t recommend network storage for the database since it’s slower.
1 Like
Ed_S
(Ed S)
February 13, 2025, 2:06pm
12
I think it’s worthwhile breaking down what your usage is. Your actual database size might not be so large, if most of your usage is uploads, and it’s only the database part which demands maybe 3x space during upgrade.
One thing you can check is the relative size of a backup with downloads compared to a backup without downloads.
Or, use the command line. Here’s some output from my rather small forum:
root@rc-debian-hel:~# free -m
total used free shared buff/cache available
Mem: 3813 1631 267 492 1915 1504
Swap: 4095 730 3365
root@rc-debian-hel:~# swapon
NAME TYPE SIZE USED PRIO
/var/local/swap/swapfile.1 file 1024M 730.2M -2
/var/local/swap/swapfile.3 file 1024M 136K -3
/var/local/swap/swapfile.0 file 1024M 0B -4
/var/local/swap/swapfile.2 file 1024M 0B -5
root@rc-debian-hel:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 382M 1.2M 381M 1% /run
/dev/sda1 38G 22G 14G 62% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 253M 6.3M 246M 3% /boot/efi
overlay 38G 22G 14G 62% /var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/merged
tmpfs 382M 0 382M 0% /run/user/0
Looking closer:
root@rc-debian-hel:~# du -kx / | sort -n | tail -33
767000 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share/pnpm
767004 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share
767020 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local
795804 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse
795808 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home
833836 /var/discourse/shared/standalone/postgres_data
884648 /var/discourse/shared/standalone/uploads/default/original
978000 /usr/lib/modules
991644 /var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/diff
991664 /var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766
1025164 /var/discourse/shared/standalone/uploads/default/optimized
1146528 /usr/lib/firmware
1350496 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff
1350512 /var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554
1909816 /var/discourse/shared/standalone/uploads/default
1919380 /var/discourse/shared/standalone/uploads
2471968 /usr/lib
2506940 /var/log/journal/82e4cf9bff9748d090b41e6d336353eb
2515140 /var/log/journal
2592200 /var/log
3029428 /usr
3839148 /var/discourse/shared/standalone/backups/default
3839152 /var/discourse/shared/standalone/backups
4194324 /var/local/swap
4194328 /var/local
5171844 /var/lib/docker/overlay2
5217052 /var/lib/docker
5455612 /var/lib
6682972 /var/discourse/shared/standalone
6682976 /var/discourse/shared
6685716 /var/discourse
19041368 /var
23037264 /
root@rc-debian-hel:~# du -kx /var/discourse/shared/ | sort -n | tail -33
42312 /var/discourse/shared/standalone/uploads/default/original/2X/f
42448 /var/discourse/shared/standalone/uploads/default/original/2X/1
42548 /var/discourse/shared/standalone/uploads/default/original/2X/6
43380 /var/discourse/shared/standalone/uploads/default/optimized/2X/2
44148 /var/discourse/shared/standalone/uploads/default/optimized/2X/5
44340 /var/discourse/shared/standalone/uploads/default/optimized/2X/1
45240 /var/discourse/shared/standalone/uploads/default/optimized/2X/e
46648 /var/discourse/shared/standalone/uploads/default/optimized/2X/c
49516 /var/discourse/shared/standalone/uploads/default/optimized/2X/8
49772 /var/discourse/shared/standalone/uploads/default/optimized/2X/9
49932 /var/discourse/shared/standalone/log/var-log/nginx
50788 /var/discourse/shared/standalone/uploads/default/optimized/2X/0
55428 /var/discourse/shared/standalone/uploads/default/optimized/2X/d
55844 /var/discourse/shared/standalone/uploads/default/optimized/2X/f
57548 /var/discourse/shared/standalone/uploads/default/optimized/2X/6
77280 /var/discourse/shared/standalone/log/var-log
81928 /var/discourse/shared/standalone/postgres_data/pg_wal
84064 /var/discourse/shared/standalone/log
294384 /var/discourse/shared/standalone/uploads/default/optimized/1X
325068 /var/discourse/shared/standalone/uploads/default/original/1X
559576 /var/discourse/shared/standalone/uploads/default/original/2X
724684 /var/discourse/shared/standalone/postgres_data/base/16384
730776 /var/discourse/shared/standalone/uploads/default/optimized/2X
749424 /var/discourse/shared/standalone/postgres_data/base
833836 /var/discourse/shared/standalone/postgres_data
884648 /var/discourse/shared/standalone/uploads/default/original
1025164 /var/discourse/shared/standalone/uploads/default/optimized
1909816 /var/discourse/shared/standalone/uploads/default
1919380 /var/discourse/shared/standalone/uploads
3839148 /var/discourse/shared/standalone/backups/default
3839152 /var/discourse/shared/standalone/backups
6682972 /var/discourse/shared/standalone
6682976 /var/discourse/shared/
2 Likes
helmi
(Helmi)
February 13, 2025, 2:21pm
13
I should have been more precise initially but didn’t expect to get such widespread feedback.
Our uploads and backups are on S3. The database size on the live system as at around 230GB now. Compressed backups are around 25GB I think.
4 Likes
Falco
(Falco)
February 13, 2025, 4:16pm
14
Wow that is one of the big ones! At that size the database operation is usually a bit more troublesome indeed.
2 Likes
pfaffman
(Jay Pfaffman)
February 14, 2025, 1:33am
15
Yow. Have you done a vacuum lately?
helmi
(Helmi)
February 14, 2025, 8:12am
16
Nope, I was under the impression that autovacuum should take care about it? No?
pfaffman
(Jay Pfaffman)
February 14, 2025, 6:39pm
17
I think it’s supposed to. I am not clear what triggers it. I think doing an extra one doesn’t hurt and might save you some space. It’s recommended after you do the upgrade if you do it in place. I’ve seen it clean up considerable space a few times.
If your database is 230GB, I’d definitely restore it to a new server. Downtime reading and writing 230GB is going to be significant.
1 Like