Make (temporary) use of Network Storage for Restores, PSQL Update,

helmi · February 12, 2025, 12:30pm

We’re running a bigger Discourse installation on a VPS setup that basically works very well for us. In terms of CPU/Mem performance we have headroom. Disk space though is a bit of an issue - not in the day to day business but when it comes to upgrading Postgres for example (the 13->15 Upgrade is outstanding because of that) we lack the space and can’t easily expand.

I know there are other options for the postgres update but consider this more of a general question.

We’re running on Hetzner where Network storage is easily available for temporary use.

On our test server i’m now playing around with making it work in a temporary way - first to restore a backup from the live site, later to test the Postgres Upgrade. So far I haven’t been successful.

I already tried symlinking but noticed it didn’t work and also read here somewhere that it’s not the recommended way. I have also tried moving the /shared share from /var/discourse/shared/standalone to /mnt/ext-storage/standalone and moved the files over there - unfortunately not without issues. I can’t even finish the build.

Is there any way that works for these kind of cases? I’m aware that the drive performance is much worse than the local drive but I’m not planning to run the forum on it. I’d really just like to have a comfortable way to use it for certain scenarios.

pfaffman · February 12, 2025, 2:31pm

If your goal is to do the upgrade the easiest thing is to spin up a new vm and migrate to it. You skip the need to upgrade the database and you get a new os on your vm, which you likely need to do anyway.

Follow Move a Discourse site to another VPS with rsync and don’t copy the database (but yes uploads and let’s encrypt and certs.

If your backups are on s3 it’s really simple to freeze the old one, make a backup, and restore the backup on the new machine.

If hetzner has some kind of permanent ip that can be assigned to different servers, you don’t even need to change dns.

You want to know that you can build a new server so that if you ever have to for some reason you’ll be able to. This is a prefect chance to practice.

helmi · February 12, 2025, 2:41pm

Actually this is not an option. It’s not running out of space for day to day needs anyways. Also we’re on a 600gb drive space right now and using ~50%. There is not a bigger option - at least not on Hetzner.

That’s why i explicitly was asking for the external drive.

merefield · February 12, 2025, 2:59pm

Have you performed a ./launcher cleanup app out of interest? Has that not released enough space to perform the upgrade in place?

helmi · February 12, 2025, 3:00pm

should that matter for a rebuild? For a simple restart I’d understand but for a rebuild?

pfaffman · February 12, 2025, 3:01pm

I wasn’t suggesting to switch to a bigger drive, just get a new server exactly like that one. Install Discourse, restore your database.

That’s a very good question.

Yes. Every rebuild builds a new container and each of those takes up space. If you’ve never done it, you could have tens of GB freed.

merefield · February 12, 2025, 3:01pm

For the database upgrade you need all the space you can get. So yes. I would say important.

Falco · February 12, 2025, 3:49pm

Our update guide includes a guide for your exact use case

PostgreSQL 15 update

Doing a manual update / space constrained environments

If you are in a constrained space environment without any way to get more space you can try the following:

./launcher stop app #(or both web_only and data if that is your case)
mkdir -p /var/discourse/shared/standalone/postgres_data_new
docker run --rm \
	--entrypoint=/bin/bash \
	-v /var/discourse/shared/standalone/postgres_data:/var/lib/postgresql/13/data \
	-v /var/discourse/shared/standalone/postgres_data_new:/var/lib/postgresql/15/data \
	tianon/postgres-upgrade:13-to-15 \
	-c "apt-get update && apt-get install -y postgresql-13-pgvector postgresql-15-pgvector &&
	docker-upgrade"
mv /var/discourse/shared/standalone/postgres_data /var/discourse/shared/standalone/postgres_data_old
mv /var/discourse/shared/standalone/postgres_data_new /var/discourse/shared/standalone/postgres_data
docker run --rm -v /var/discourse/shared/standalone:/shared \
local_discourse/app chown -R postgres:postgres /shared/postgres_data #(or local_discourse/data)
./launcher rebuild app #(or first data and then web_only if that is your case)

On my tests this procedure requires less than 1x your current database size in free space.

We added this option for people who in the same situation as you. Just be sure to store a backup offsite before trying this!

pfaffman · February 12, 2025, 6:11pm

If you’re on a VM, it’s much, much easier to just move to a new one, and there are many benefits. Here are a few:

zero risk–if something goes wrong, you still have your old server
zero downtime (just read-only on the old server)
you get the OS upgrade that you probably need anyway
you can verify that you know how to spin up a new server should calamity strike
you don’t need to reindex and vacuum

helmi · February 13, 2025, 8:29am

Appreciate your advise guys.

Thanks, thats the other option that I said I’m aware of. Thanks anyways for pointing it out.

Not doubting that - still would have loved to explore the option of adding additional storage for maintenance tasks if needed.

pfaffman · February 13, 2025, 10:20am

It can be handy to keep your uploads, or, say /var/discourse/shared/web_only on network storage. You need to edit the yml file to point to it rather than use a symlink (the symlink doesn’t work because the container can’t access the place that your symlink points to.

Then if you do move to a new vm, you can just remount that network storage rather than copy it over.

I don’t recommend network storage for the database since it’s slower.

Ed_S · February 13, 2025, 2:06pm

I think it’s worthwhile breaking down what your usage is. Your actual database size might not be so large, if most of your usage is uploads, and it’s only the database part which demands maybe 3x space during upgrade.

One thing you can check is the relative size of a backup with downloads compared to a backup without downloads.

Or, use the command line. Here’s some output from my rather small forum:

root@rc-debian-hel:~# free -m
               total        used        free      shared  buff/cache   available
Mem:            3813        1631         267         492        1915        1504
Swap:           4095         730        3365
root@rc-debian-hel:~# swapon 
NAME                       TYPE  SIZE   USED PRIO
/var/local/swap/swapfile.1 file 1024M 730.2M   -2
/var/local/swap/swapfile.3 file 1024M   136K   -3
/var/local/swap/swapfile.0 file 1024M     0B   -4
/var/local/swap/swapfile.2 file 1024M     0B   -5
root@rc-debian-hel:~# df -h 
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           382M  1.2M  381M   1% /run
/dev/sda1        38G   22G   14G  62% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      253M  6.3M  246M   3% /boot/efi
overlay          38G   22G   14G  62% /var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/merged
tmpfs           382M     0  382M   0% /run/user/0

Looking closer:

root@rc-debian-hel:~# du -kx / | sort -n | tail -33
767000	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share/pnpm
767004	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local/share
767020	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse/.local
795804	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home/discourse
795808	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff/home
833836	/var/discourse/shared/standalone/postgres_data
884648	/var/discourse/shared/standalone/uploads/default/original
978000	/usr/lib/modules
991644	/var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766/diff
991664	/var/lib/docker/overlay2/68abab42f48040e0dfc03d3c9fc893dfa3e7fb01ba1b2215731668339bbc3766
1025164	/var/discourse/shared/standalone/uploads/default/optimized
1146528	/usr/lib/firmware
1350496	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554/diff
1350512	/var/lib/docker/overlay2/8b6ac2d69a1fa195285e61aba2876484a69fd2a19032c2f4def1c4adb02d6554
1909816	/var/discourse/shared/standalone/uploads/default
1919380	/var/discourse/shared/standalone/uploads
2471968	/usr/lib
2506940	/var/log/journal/82e4cf9bff9748d090b41e6d336353eb
2515140	/var/log/journal
2592200	/var/log
3029428	/usr
3839148	/var/discourse/shared/standalone/backups/default
3839152	/var/discourse/shared/standalone/backups
4194324	/var/local/swap
4194328	/var/local
5171844	/var/lib/docker/overlay2
5217052	/var/lib/docker
5455612	/var/lib
6682972	/var/discourse/shared/standalone
6682976	/var/discourse/shared
6685716	/var/discourse
19041368	/var
23037264	/
root@rc-debian-hel:~# du -kx /var/discourse/shared/ | sort -n | tail -33
42312	/var/discourse/shared/standalone/uploads/default/original/2X/f
42448	/var/discourse/shared/standalone/uploads/default/original/2X/1
42548	/var/discourse/shared/standalone/uploads/default/original/2X/6
43380	/var/discourse/shared/standalone/uploads/default/optimized/2X/2
44148	/var/discourse/shared/standalone/uploads/default/optimized/2X/5
44340	/var/discourse/shared/standalone/uploads/default/optimized/2X/1
45240	/var/discourse/shared/standalone/uploads/default/optimized/2X/e
46648	/var/discourse/shared/standalone/uploads/default/optimized/2X/c
49516	/var/discourse/shared/standalone/uploads/default/optimized/2X/8
49772	/var/discourse/shared/standalone/uploads/default/optimized/2X/9
49932	/var/discourse/shared/standalone/log/var-log/nginx
50788	/var/discourse/shared/standalone/uploads/default/optimized/2X/0
55428	/var/discourse/shared/standalone/uploads/default/optimized/2X/d
55844	/var/discourse/shared/standalone/uploads/default/optimized/2X/f
57548	/var/discourse/shared/standalone/uploads/default/optimized/2X/6
77280	/var/discourse/shared/standalone/log/var-log
81928	/var/discourse/shared/standalone/postgres_data/pg_wal
84064	/var/discourse/shared/standalone/log
294384	/var/discourse/shared/standalone/uploads/default/optimized/1X
325068	/var/discourse/shared/standalone/uploads/default/original/1X
559576	/var/discourse/shared/standalone/uploads/default/original/2X
724684	/var/discourse/shared/standalone/postgres_data/base/16384
730776	/var/discourse/shared/standalone/uploads/default/optimized/2X
749424	/var/discourse/shared/standalone/postgres_data/base
833836	/var/discourse/shared/standalone/postgres_data
884648	/var/discourse/shared/standalone/uploads/default/original
1025164	/var/discourse/shared/standalone/uploads/default/optimized
1909816	/var/discourse/shared/standalone/uploads/default
1919380	/var/discourse/shared/standalone/uploads
3839148	/var/discourse/shared/standalone/backups/default
3839152	/var/discourse/shared/standalone/backups
6682972	/var/discourse/shared/standalone
6682976	/var/discourse/shared/

helmi · February 13, 2025, 2:21pm

I should have been more precise initially but didn’t expect to get such widespread feedback.

Our uploads and backups are on S3. The database size on the live system as at around 230GB now. Compressed backups are around 25GB I think.

Falco · February 13, 2025, 4:16pm

Wow that is one of the big ones! At that size the database operation is usually a bit more troublesome indeed.

pfaffman · February 14, 2025, 1:33am

Yow. Have you done a vacuum lately?

helmi · February 14, 2025, 8:12am

Nope, I was under the impression that autovacuum should take care about it? No?

pfaffman · February 14, 2025, 6:39pm

I think it’s supposed to. I am not clear what triggers it. I think doing an extra one doesn’t hurt and might save you some space. It’s recommended after you do the upgrade if you do it in place. I’ve seen it clean up considerable space a few times.

If your database is 230GB, I’d definitely restore it to a new server. Downtime reading and writing 230GB is going to be significant.

Topic		Replies	Views
Stuck in a loop of freeing up space and filling it up again when rebuilding Installation server-resources	33	3643	June 8, 2024
PostgreSQL Stuck During Rebuilding Installation	52	706	October 4, 2024
Upgrade failed due to unclean database shutdown Installation	21	4171	November 9, 2017
Docker base image with PostgreSQL 10 released Announcements docker	26	4672	December 1, 2023
Upgrade Postgres with REALLY limited space Installation server-resources	17	2355	February 21, 2022

Make (temporary) use of Network Storage for Restores, PSQL Update,

Related topics