Unable to rebuild discourse due to no space left on device

Kind of unavoidable I think we bumped base image 2 times in the last few months. Not much we can do. Looks like the cleanup on your server shaved off 2 base images.

3 Likes

@anon43908006, there is a guide at:

It goes over many considerations for changing your domain, check it out. :slight_smile:

1 Like

To clarify, is there not much to do about the overall size of the upgrades increasing, or not much to do about the recent spike in bumping base image activity (which will not affect as much in the future)?

I’v been surprised, I’ve got all these tiny Discourses with very few users, and kept running into this issue lately. No uploads or anything. I was wondering if we are approaching a point where the cloud install will recommend the next size up on drive space (which is 2GB RAM/1vCPU/50GB SSD). :thinking:

5 Likes

I asked @falco about this in chat and he said lately we’ve had a lot of base image changes due to dependencies being updated, so there’s been a higher than usual amount of disk space in upgrades in the last ~6 months.

5 Likes

Sorry to hear you’ve run into trouble changing your domain name, @anon43908006.

Because this is #support, I encourage you to create a new topic that explains your exact case: it may be that your situation requires more discussion than what’s happening in this topic, which is more of a general pattern we’ve been noticing.

If you’d like, you can mention me (@maiki) and I’d be happy to discuss what’s going on with your site. :slight_smile:

6 Likes

I get the same No space left on device error when I try to backup my Discourse:

[2022-11-15 08:23:38] EXCEPTION: /var/www/discourse/lib/discourse.rb:131:in `exec': Failed to gzip archive.

gzip: /var/www/discourse/public/backups/default/forum-leasehackr-2022-11-15-080439-v20221110175456.tar.gz: No space left on device

My backup and image uploads are set up on DigitalOcean’s Spaces and have been working fine for a few years until recent months. Here are what I have tried so far:

  1. I cleared all hidden multipart uploads on my DO Space. There should be more than 100GiB of storage available on my DO Space.
  2. I tried rebuilding and cleaning up using the following commands:
cd /var/discourse
apt-get update
apt-get upgrade
apt-get autoclean
apt-get autoremove
./launcher rebuild app
./launcher cleanup

Anyone knows why my backups keep failing still? Thank you!

Check to see if you have leftover backups here and delete them:
/var/discourse/shared/standalone/backups/default

2 Likes

This message is about your local disk. To create a backup you need to have almost twice the size of your database+uploads available to create the backup that gets uploaded to spaces. What do these show you?

 df -h 
 du -hs /var/discourse/shared/*
 du -hs /var/discourse/shared/standalone

show you?

4 Likes

This is what the commands show:

root@forum:~# df -h 
Filesystem                 Size  Used Avail Use% Mounted on
udev                       7.9G     0  7.9G   0% /dev
tmpfs                      1.6G  169M  1.5G  11% /run
/dev/disk/by-label/DOROOT  315G  277G   22G  93% /
tmpfs                      7.9G  4.9M  7.9G   1% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                      7.9G     0  7.9G   0% /sys/fs/cgroup
cgmfs                      100K     0  100K   0% /run/cgmanager/fs
none                       315G  277G   22G  93% /var/lib/docker/aufs/mnt/990236bd36313269b6683947dc21241f4387b2671676e8fdf68de05c0fb3e0f4
shm                        512M   22M  491M   5% /var/lib/docker/containers/4684612f16f4f31589ae8075145bcde0e6e9576a22eaef3952e9d47986999b91/shm
tmpfs                      1.6G     0  1.6G   0% /run/user/0
root@forum:~# du -hs /var/discourse/shared/*
215G    /var/discourse/shared/standalone
root@forum:~# du -hs /var/discourse/shared/standalone
215G    /var/discourse/shared/standalone

If it’s a local storage issue, can I simply add block storage (Volumes) to the DigitcalOcean Droplet?

Thank you!

There is a topic about that. I would likely move backups to Spaces first.

What about this

   du -hs /var/discourse/shared/*

I have backups set up on DO Spaces. I thought the backup failed due to not having enough space locally?

Oops

 du -hs /var/discourse/shared/standalone/*

Oh. Yes, local storage is the problem, but an easy way to solve it is moving backups, but you did that already…

But maybe you can do a

 ./launcher cleanup

And get back space from docker

This is the output:

root@forum:~# du -hs /var/discourse/shared/standalone/*
124G    /var/discourse/shared/standalone/backups
21M     /var/discourse/shared/standalone/letsencrypt
2.9G    /var/discourse/shared/standalone/log
4.0K    /var/discourse/shared/standalone/postgres_backup
74G     /var/discourse/shared/standalone/postgres_data
264K    /var/discourse/shared/standalone/postgres_run
5.1G    /var/discourse/shared/standalone/redis_data
44K     /var/discourse/shared/standalone/ssl
28K     /var/discourse/shared/standalone/state
16K     /var/discourse/shared/standalone/tmp
9.2G    /var/discourse/shared/standalone/uploads

I ran ./launcher cleanup two days ago and it didn’t solve the backup fail problem. :confused:

You have too much storage being consumed by your backups. You’ll need to rethink your backup strategy and possibly move backups to an external storage like S3.

1 Like

Either you don’t have backups on spaces or you didn’t delete the local backups when you switched to it. You can delete those backups and be in good shape.

2 Likes

I believe it’s the latter because I do see my backup files on my DO Spaces. What is the best practice to delete the backups? Can I remove the content with the following command?

rm -r /var/discourse/shared/standalone/backups

Thank you!

1 Like

That should do the trick!

1 Like

That was indeed the cause. Just cleared out the local backup folder and backup works again. Thank you for your help!

1 Like

Hooray! With 126Gb free, you should be good for a while!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.