Discourse upgrade fails due to insufficient disk space on 25G droplet

installation
21

18G /var

du -h -s /var/* | sort -h -r
14G     /var/lib
2.8G    /var/log
933M    /var/discourse

du -h -s /var/lib/* | sort -h -r
13G     /var/lib/docker
744M    /var/lib/snapd

root@DO-Discourse:/var/discourse# du -h -s /var/lib/docker/* | sort -h -r
13G     /var/lib/docker/overlay2
16M     /var/lib/docker/image

root@DO-Discourse:/var/discourse# du -h -s /var/lib/docker/overlay2/* | sort -h -r
8.7G    /var/lib/docker/overlay2/d319d95263d87c2a75a4bc9a9f03a25ea7f6eb1f7bac687e7ae7d45522939dc0
2.8G    /var/lib/docker/overlay2/79be56509f1588c272683332ef50abd54f0aeb06d0e2d13f8eea1bace3b3db46
873M    /var/lib/docker/overlay2/5b148cbbcca894be512c7407568104cd7b2e3d48ab7b7d74c6c0f731806cdddc

Any point going further?

22

Deleted 2.8G of logs, which only got me to 4.9G. Don’t have a “test” instance to try ./launcher rebuild app --skip-prereqs

Any other suggestions?

Is it simply not possible to run a discourse instance on a 25G droplet anymore?

23

See the end of Prune unused Docker objects | Docker Documentation

I think you want to prune all and prune volumes

24 
docker system prune --volumes
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all volumes not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
root@DO-Discourse:/#

Any other suggestions?

25

do you have any use for snap ?

26

Not really, that’s pretty telling that the space is being consumed by docker for your container(s). After a cleanup, my numbers are very similar to yours which is perhaps (but not necessarily) indicative of it just being how much it uses.

I did have problems with a 25GB Linode however that was with 500MB+ backups, deleting two or three backups gave me enough room to rebuild. I opted to move to the next tier with 50GB because it was only going to become more constraining and I wanted to rebuild on a monthly cron job.

That was before the switch to Ember CLI though, could that have made things significantly bigger?

27

That 9GB overlay seems like the issue. But I have one on of a similar size on another instance I just checked. It’s always been challenging at 25GB. I’d recommend that you bite the bullet and get more SSD. The next thing you could try would be to see if there is OS-level stuff that you can remove (logs, unneeded programs, indexes from find, maybe?).

Another idea would be to just spin up a new 25GB VM and move there, hoping that whatever filled up the old one won’t be a problem this time.

None of these answers seems especially satisfying. I fought hard with a 25GB droplet on one or two instances I help manage in the last week or two, but I think you’ve done everything that I did.

3 Likes
28

Not sure. Do I need it for a discourse-only installation? And if not, how do I delete it?

29

I don’t think so :thinking:
after a full backup download, you could snap list to check what snaps are installed, and if none sudo apt purge snapd

30 
root@DO-Discourse:/var/discourse# snap list
Name    Version        Rev    Tracking       Publisher   Notes
core18  20220309       2344   latest/stable  canonical✓  base
core20  20220304       1376   latest/stable  canonical✓  base
lxd     4.0.9-8e2046b  22753  4.0/stable/…   canonical✓  -
snapd   2.54.4         15177  latest/stable  canonical✓  snapd
root@DO-Discourse:/var/discourse#
31

What are those overlays? Can I delete them?

32

Hi Andy, sorry for the delay.
At this point please keep in mind that without insight of your host system, it’s easy to guide you toward a cliff, so backup/snapshot/etc… (or even take a backup, spin up a fresh instance and test the restore process, it has been a good exercice for me)
Assuming you have no use for lxd (lxc list should show installed containers) snap remove lxd (and then core18 and 20)

1 Like
33

Can you share the output of docker images and docker ps -a ?

34

Thank you for your help. Yes, I know so little about Linux, so I fully appreciate the advice to backup. I have an automatic snapshot and do a manual one whenever I’m messing around like now.

Have no idea what lxd is. I don’t need anything on this droplet that discourse/docker doesn’t need, as it’s solely a discourse droplet.

1 Like
35 
root@DO-Discourse:/var/discourse# docker images
REPOSITORY            TAG       IMAGE ID       CREATED        SIZE
local_discourse/app   latest    3dac608caa92   4 months ago   3.17GB
root@DO-Discourse:/var/discourse#

root@DO-Discourse:/var/discourse# docker ps -a
CONTAINER ID   IMAGE                 COMMAND        CREATED        STATUS        PORTS                                      NAMES
9abaf4517b7e   local_discourse/app   "/sbin/boot"   4 months ago   Up 4 months   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   app
root@DO-Discourse:/var/discourse#
36

Considering

and

I think you have some dangling layers on your overlay2 folder.

See this StackOverflow answer for guidance:

3 Likes
37

Thank you.
That’s quite a bit over my head, would it be safe to just do a snapshot and then delete them and see what happens?

1 Like
38

I think it’s safe to delete all of those images. If you need them they’ll get replaced when you rebuild.

Taking a snapshot isn’t a bad idea.

What I’d do it just spin up a new instance; that’s safest, and probably faster than a snapshot, but if knowing how to do that isn’t fun or instructive, your idea is fine.

2 Likes
39

Deleted one of the images and that did the trick!

Thank you everyone for all your help.

2 Likes
40

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.