Unable to rebuild discourse due to no space left on device

I get the aforementioned error while attempting to do an operation. I have no idea as to why this is happening.

1 Like

Did you try just doing a

./docker rebuild app

and seeing what happens? I think git pull is no longer required first.

Otherwise it looks like you may need to review your app.conf file. Have you edited it recently?

1 Like

No I haven’t edited it recently. The website crashed yesterday and I ran the cleanup and then ran
rm /var/discourse/shared/standalone/backups/default/*
Then I rebuilt using ./launcher rebuild app

The website started working again after that and now it’s back to being dead.

1 Like

Sorry I meant

./launcher rebuild app

So you are doing the the right things.

1 Like

Have you had a look at Discourse Doctor?

1 Like

Okay, so it is a storage issue. How do I make space now? I’m sorry but I’m a beginner.

I just ran discourse-doctor and I was left with multiple lines stating that my storage was full.

1 Like

Do you have anything else on the server? If not, it’s probably discourse backups that you can delete.

1 Like

Can you go over the process of deleting backups coz I’ve never really understood the process. I wanna be sure once and for all because I’ve been having storage issues for a really long time.

No, I don’t have anything else on the server.

1 Like

A good first step is to run

./launcher cleanup 

If that doesn’t work, try

./discourse-doctor

If you still have difficulties, you can look at deleting old backups from

/var/discourse/shared/standalone/backups/default

Let us know how these work out for you!

4 Likes

Hi @seshu_ram

Often, when containers are rebuilt, the process leaves orphan images. If you have rebuilt your container often, these images can take up a lot of space.

In fact, these orphan images recently took up nearly 100 GB + on our server until I deleted them. You can easily check.

Please post the output of:

docker images

Kindly post the output as text (copy-and-paste) using fenced markdown. Terminal screenshot images are hard to read on mobile.

Thanks.

Note:

Please note that launcher cleanup also prunes these orphans (based on 24 hours in the past, I think):

if tty >/dev/null; then
      read -p "Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)" -n 1 -r
      echo
      if [[ $REPLY =~ ^[Yy]$ ]]
      then
        $docker_path container prune --force --filter until=1h >/dev/null
        $docker_path image prune --all --force --filter until=1h >/dev/null
        echo "If the cleanup was successful, you may try again now"
      fi
    fi
3 Likes

@neounix

local_discourse/app   latest              674fd54f165f        4 minutes ago       2.5GB
<none>                <none>              f3a4104c3f75        22 hours ago        2.5GB
discourse/base        2.0.20201221-2020   c0704d4ce2b4        11 days ago         2.11GB ```
1 Like

This worked. My website is live now. Thank you so much. Thanks a lot for your time! That helped a lot.

@tobiaseigen

4 Likes

Hey @seshu_ram

FYI and FWIW: You can remove this orphan image and reclaim a bit more disk space:

 f3a4104c3f75   
docker image rm  f3a4104c3f75  

The launcher cleanup process does not (as I recall) remove images less than 24 hours old.

Or, you can run cleanup again in a few hours, as you please.

5 Likes

One thing I’ve noticed is that recent command line updates of Discourse are eating quite a bit of disk space…

root@endoffice-b:/var/discourse# ./launcher cleanup
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] Y
Total reclaimed space: 0B
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] Y
Deleted Images:
deleted: sha256:284403a252ba061b3ab97f4bfe293ac5e8f05f39ada429d718f58e56191251c2
deleted: sha256:6b6899d54d4dd1f21568956b652975f7c0b9e439978b8cc53036efc46baaf971
untagged: discourse/base:2.0.20211118-0105
untagged: discourse/base@sha256:74b41fffd4f05433eb7c9b72954b1f5f8b15cd0e802bb724c96b7d699c3f6fa1
deleted: sha256:b6cc7cf8974a6ef7bb64c36f4592af261cda0d5565bd91da603568ce26968048
deleted: sha256:c1455b2fdbca024c36c4e75746051b77c3637020cfa1e36a41440292a8c39424
deleted: sha256:77b323d4ec74aad770337f99a60e862a64ccc53f4775b5f4945df0e606f78b90
untagged: discourse/base:2.0.20220128-1817
untagged: discourse/base@sha256:dcb4eb8e41a2e84f776f80587f308d167a54ad7ff4ba616199891828bbd4ddae

Total reclaimed space: 3.54GB

This happened on both instances, the other was 3.538GB :wink:

I’m usually pretty religious about running ./launcher cleanup after each Discourse update, and I update about once a month, so that says to me the last update by itself ate nearly 4GB of disk… cc @falco @sam is this something we should be concerned about? :thinking:

4 Likes

Kind of unavoidable I think we bumped base image 2 times in the last few months. Not much we can do. Looks like the cleanup on your server shaved off 2 base images.

3 Likes

I have the same issue and am also a beginner and non-linux user.

I really can’t understand the technical limitations of the site.

Apparently I have ‘only’ 5GB free (80%) and am trying to rebuild the site without success.

/ cleanup has been run several times.

The forum is exceptionally small and new. I’m really just trying to reinstall an SSL certificate since changing our main domain URL.

Whats the easy way to rebuild a discourse server and reinstall letsencrypt please?

1 Like

@bizinbarefeet, there is a guide at:

It goes over many considerations for changing your domain, check it out. :slight_smile:

1 Like

To clarify, is there not much to do about the overall size of the upgrades increasing, or not much to do about the recent spike in bumping base image activity (which will not affect as much in the future)?

I’v been surprised, I’ve got all these tiny Discourses with very few users, and kept running into this issue lately. No uploads or anything. I was wondering if we are approaching a point where the cloud install will recommend the next size up on drive space (which is 2GB RAM/1vCPU/50GB SSD). :thinking:

4 Likes

I asked @falco about this in chat and he said lately we’ve had a lot of base image changes due to dependencies being updated, so there’s been a higher than usual amount of disk space in upgrades in the last ~6 months.

5 Likes

Thanks for the direction. Must admin it has not solved my problem. Another day gone on Discourse - sigh.

Rebuild failed. While the ./discourse-setup function works it also installs a database that is 3 months old. Restoring an image changes the domain back to the old one but does restore the old posts.

Think I’ll have to google how to edit this app.yml file after restoring the old posts (again).

Well that failed too. The app has lost 3 months worth of posts. Strangely the old and new domain names are pointed at exactly the same droplet thingy. But show different content.

The old domain has the latest posts, although the posts aren’t clickable. And the new app.yml edited one doesn’t show the latest posts.

Thanks for the help offered by life is too short of Discourse. We’ll be moving onto something else.