Unable to rebuild discourse due to no space left on device

I get the aforementioned error while attempting to do an operation. I have no idea as to why this is happening.

2 Likes

Did you try just doing a

./docker rebuild app

and seeing what happens? I think git pull is no longer required first.

Otherwise it looks like you may need to review your app.conf file. Have you edited it recently?

1 Like

No I haven’t edited it recently. The website crashed yesterday and I ran the cleanup and then ran
rm /var/discourse/shared/standalone/backups/default/*
Then I rebuilt using ./launcher rebuild app

The website started working again after that and now it’s back to being dead.

1 Like

Sorry I meant

./launcher rebuild app

So you are doing the the right things.

1 Like

Have you had a look at Discourse Doctor?

1 Like

Okay, so it is a storage issue. How do I make space now? I’m sorry but I’m a beginner.

I just ran discourse-doctor and I was left with multiple lines stating that my storage was full.

1 Like

Do you have anything else on the server? If not, it’s probably discourse backups that you can delete.

1 Like

Can you go over the process of deleting backups coz I’ve never really understood the process. I wanna be sure once and for all because I’ve been having storage issues for a really long time.

No, I don’t have anything else on the server.

1 Like

A good first step is to run

./launcher cleanup 

If that doesn’t work, try

./discourse-doctor

If you still have difficulties, you can look at deleting old backups from

/var/discourse/shared/standalone/backups/default

Let us know how these work out for you!

4 Likes

Hi @seshu_ram

Often, when containers are rebuilt, the process leaves orphan images. If you have rebuilt your container often, these images can take up a lot of space.

In fact, these orphan images recently took up nearly 100 GB + on our server until I deleted them. You can easily check.

Please post the output of:

docker images

Kindly post the output as text (copy-and-paste) using fenced markdown. Terminal screenshot images are hard to read on mobile.

Thanks.

Note:

Please note that launcher cleanup also prunes these orphans (based on 24 hours in the past, I think):

if tty >/dev/null; then
      read -p "Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)" -n 1 -r
      echo
      if [[ $REPLY =~ ^[Yy]$ ]]
      then
        $docker_path container prune --force --filter until=1h >/dev/null
        $docker_path image prune --all --force --filter until=1h >/dev/null
        echo "If the cleanup was successful, you may try again now"
      fi
    fi
3 Likes

@neounix

local_discourse/app   latest              674fd54f165f        4 minutes ago       2.5GB
<none>                <none>              f3a4104c3f75        22 hours ago        2.5GB
discourse/base        2.0.20201221-2020   c0704d4ce2b4        11 days ago         2.11GB ```
1 Like

This worked. My website is live now. Thank you so much. Thanks a lot for your time! That helped a lot.

@tobiaseigen

4 Likes

Hey @seshu_ram

FYI and FWIW: You can remove this orphan image and reclaim a bit more disk space:

 f3a4104c3f75   
docker image rm  f3a4104c3f75  

The launcher cleanup process does not (as I recall) remove images less than 24 hours old.

Or, you can run cleanup again in a few hours, as you please.

5 Likes

One thing I’ve noticed is that recent command line updates of Discourse are eating quite a bit of disk space…

root@endoffice-b:/var/discourse# ./launcher cleanup
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] Y
Total reclaimed space: 0B
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] Y
Deleted Images:
deleted: sha256:284403a252ba061b3ab97f4bfe293ac5e8f05f39ada429d718f58e56191251c2
deleted: sha256:6b6899d54d4dd1f21568956b652975f7c0b9e439978b8cc53036efc46baaf971
untagged: discourse/base:2.0.20211118-0105
untagged: discourse/base@sha256:74b41fffd4f05433eb7c9b72954b1f5f8b15cd0e802bb724c96b7d699c3f6fa1
deleted: sha256:b6cc7cf8974a6ef7bb64c36f4592af261cda0d5565bd91da603568ce26968048
deleted: sha256:c1455b2fdbca024c36c4e75746051b77c3637020cfa1e36a41440292a8c39424
deleted: sha256:77b323d4ec74aad770337f99a60e862a64ccc53f4775b5f4945df0e606f78b90
untagged: discourse/base:2.0.20220128-1817
untagged: discourse/base@sha256:dcb4eb8e41a2e84f776f80587f308d167a54ad7ff4ba616199891828bbd4ddae

Total reclaimed space: 3.54GB

This happened on both instances, the other was 3.538GB :wink:

I’m usually pretty religious about running ./launcher cleanup after each Discourse update, and I update about once a month, so that says to me the last update by itself ate nearly 4GB of disk… cc @falco @sam is this something we should be concerned about? :thinking:

4 Likes

Kind of unavoidable I think we bumped base image 2 times in the last few months. Not much we can do. Looks like the cleanup on your server shaved off 2 base images.

3 Likes

@anon43908006, there is a guide at:

It goes over many considerations for changing your domain, check it out. :slight_smile:

1 Like

To clarify, is there not much to do about the overall size of the upgrades increasing, or not much to do about the recent spike in bumping base image activity (which will not affect as much in the future)?

I’v been surprised, I’ve got all these tiny Discourses with very few users, and kept running into this issue lately. No uploads or anything. I was wondering if we are approaching a point where the cloud install will recommend the next size up on drive space (which is 2GB RAM/1vCPU/50GB SSD). :thinking:

5 Likes

I asked @falco about this in chat and he said lately we’ve had a lot of base image changes due to dependencies being updated, so there’s been a higher than usual amount of disk space in upgrades in the last ~6 months.

5 Likes

Sorry to hear you’ve run into trouble changing your domain name, @anon43908006.

Because this is support, I encourage you to create a new topic that explains your exact case: it may be that your situation requires more discussion than what’s happening in this topic, which is more of a general pattern we’ve been noticing.

If you’d like, you can mention me (@maiki) and I’d be happy to discuss what’s going on with your site. :slight_smile:

6 Likes

I get the same No space left on device error when I try to backup my Discourse:

[2022-11-15 08:23:38] EXCEPTION: /var/www/discourse/lib/discourse.rb:131:in `exec': Failed to gzip archive.

gzip: /var/www/discourse/public/backups/default/forum-leasehackr-2022-11-15-080439-v20221110175456.tar.gz: No space left on device

My backup and image uploads are set up on DigitalOcean’s Spaces and have been working fine for a few years until recent months. Here are what I have tried so far:

  1. I cleared all hidden multipart uploads on my DO Space. There should be more than 100GiB of storage available on my DO Space.
  2. I tried rebuilding and cleaning up using the following commands:
cd /var/discourse
apt-get update
apt-get upgrade
apt-get autoclean
apt-get autoremove
./launcher rebuild app
./launcher cleanup

Anyone knows why my backups keep failing still? Thank you!