Kind of unavoidable I think we bumped base image 2 times in the last few months. Not much we can do. Looks like the cleanup on your server shaved off 2 base images.
@anon43908006, there is a guide at:
It goes over many considerations for changing your domain, check it out.
To clarify, is there not much to do about the overall size of the upgrades increasing, or not much to do about the recent spike in bumping base image activity (which will not affect as much in the future)?
I’v been surprised, I’ve got all these tiny Discourses with very few users, and kept running into this issue lately. No uploads or anything. I was wondering if we are approaching a point where the cloud install will recommend the next size up on drive space (which is 2GB RAM/1vCPU/50GB SSD).
I asked @falco about this in chat and he said lately we’ve had a lot of base image changes due to dependencies being updated, so there’s been a higher than usual amount of disk space in upgrades in the last ~6 months.
Sorry to hear you’ve run into trouble changing your domain name, @anon43908006.
Because this is #support, I encourage you to create a new topic that explains your exact case: it may be that your situation requires more discussion than what’s happening in this topic, which is more of a general pattern we’ve been noticing.
If you’d like, you can mention me (@maiki) and I’d be happy to discuss what’s going on with your site.
I get the same No space left on device
error when I try to backup my Discourse:
[2022-11-15 08:23:38] EXCEPTION: /var/www/discourse/lib/discourse.rb:131:in `exec': Failed to gzip archive.
gzip: /var/www/discourse/public/backups/default/forum-leasehackr-2022-11-15-080439-v20221110175456.tar.gz: No space left on device
My backup and image uploads are set up on DigitalOcean’s Spaces and have been working fine for a few years until recent months. Here are what I have tried so far:
- I cleared all hidden multipart uploads on my DO Space. There should be more than 100GiB of storage available on my DO Space.
- I tried rebuilding and cleaning up using the following commands:
cd /var/discourse
apt-get update
apt-get upgrade
apt-get autoclean
apt-get autoremove
./launcher rebuild app
./launcher cleanup
Anyone knows why my backups keep failing still? Thank you!
Check to see if you have leftover backups here and delete them:
/var/discourse/shared/standalone/backups/default
This message is about your local disk. To create a backup you need to have almost twice the size of your database+uploads available to create the backup that gets uploaded to spaces. What do these show you?
df -h
du -hs /var/discourse/shared/*
du -hs /var/discourse/shared/standalone
show you?
This is what the commands show:
root@forum:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 169M 1.5G 11% /run
/dev/disk/by-label/DOROOT 315G 277G 22G 93% /
tmpfs 7.9G 4.9M 7.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
none 315G 277G 22G 93% /var/lib/docker/aufs/mnt/990236bd36313269b6683947dc21241f4387b2671676e8fdf68de05c0fb3e0f4
shm 512M 22M 491M 5% /var/lib/docker/containers/4684612f16f4f31589ae8075145bcde0e6e9576a22eaef3952e9d47986999b91/shm
tmpfs 1.6G 0 1.6G 0% /run/user/0
root@forum:~# du -hs /var/discourse/shared/*
215G /var/discourse/shared/standalone
root@forum:~# du -hs /var/discourse/shared/standalone
215G /var/discourse/shared/standalone
If it’s a local storage issue, can I simply add block storage (Volumes) to the DigitcalOcean Droplet?
Thank you!
There is a topic about that. I would likely move backups to Spaces first.
What about this
du -hs /var/discourse/shared/*
I have backups set up on DO Spaces. I thought the backup failed due to not having enough space locally?
Oops
du -hs /var/discourse/shared/standalone/*
Oh. Yes, local storage is the problem, but an easy way to solve it is moving backups, but you did that already…
But maybe you can do a
./launcher cleanup
And get back space from docker
This is the output:
root@forum:~# du -hs /var/discourse/shared/standalone/*
124G /var/discourse/shared/standalone/backups
21M /var/discourse/shared/standalone/letsencrypt
2.9G /var/discourse/shared/standalone/log
4.0K /var/discourse/shared/standalone/postgres_backup
74G /var/discourse/shared/standalone/postgres_data
264K /var/discourse/shared/standalone/postgres_run
5.1G /var/discourse/shared/standalone/redis_data
44K /var/discourse/shared/standalone/ssl
28K /var/discourse/shared/standalone/state
16K /var/discourse/shared/standalone/tmp
9.2G /var/discourse/shared/standalone/uploads
I ran ./launcher cleanup
two days ago and it didn’t solve the backup fail problem.
You have too much storage being consumed by your backups. You’ll need to rethink your backup strategy and possibly move backups to an external storage like S3.
Either you don’t have backups on spaces or you didn’t delete the local backups when you switched to it. You can delete those backups and be in good shape.
I believe it’s the latter because I do see my backup files on my DO Spaces. What is the best practice to delete the backups? Can I remove the content with the following command?
rm -r /var/discourse/shared/standalone/backups
Thank you!
That should do the trick!
That was indeed the cause. Just cleared out the local backup folder and backup works again. Thank you for your help!
Hooray! With 126Gb free, you should be good for a while!
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.