# Errno::ENOSPC: No space left on device

**URL:** https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829
**Category:** Support
**Tags:** docker
**Created:** [September 28, 2015, 11:06am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829 "2015-09-28T11:06:06Z")
**Posts on this page:** 18
**Page:** 1

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:06am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/1 "2015-09-28T11:06:06Z")

</div>

Hi there,

I’m trying to rebuild Discourse but I’m running into this error:

```
FAILED
--------------------
RuntimeError: cd /var/www/discourse && su discourse -c 'bundle exec rake assets:precompile' failed with return #<Process::Status: pid 4266 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:105:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
702e319df6b2f9a477ab0ad6eebdd689ad0b7bab97770d4c80fd82a73f3a95dc
**FAILED TO BOOTSTRAP** please scroll up and look for earlier error messages, there may be more than one

```

Scrolling up I found:

```
Errno::ENOSPC: No space left on device - /var/www/discourse/tmp/cache/assets/production/sprockets/c88f54c1a16e5204b2aebda1caf30657

```

I tried locating the `/var/www/discourse/` directory, but it doesn’t exist:

```
ubuntu@discourse:/var/discourse$ cd /var/www/discourse
-bash: cd: /var/www/discourse: No such file or directory

```

Any thoughts on how to recover from this?

---

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:12am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/2 "2015-09-28T11:12:19Z")

</div>

Weirdly, I seem to be out of disk space indeed:

```
ubuntu@discourse:/var/discourse$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 6.7G 700M 91% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 996M 12K 996M 1% /dev
tmpfs 201M 356K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user

```

How did this happen? This is a new install with less than 50 topics.

I have built and rebuilt the app a few times; could that be reason I ran out of disk space? If so, then the question is: why? Could this be because of temporary files that were not being deleted?

I thought `/var/www/discourse` was a good place to start, but it seems like it isn’t.

Where should I look?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 28, 2015, 11:15am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/3 "2015-09-28T11:15:37Z")

</div>

can you start with a

```plaintext
./launcher cleanup

```

I bet you have 2 docker images going

---

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:22am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/4 "2015-09-28T11:22:57Z")

</div>

Thanks, @sam, but that unfortunately didn’t solve the problem:

```
ubuntu@discourse:/var$ cd /var/discourse/
ubuntu@discourse:/var/discourse$ sudo ./launcher cleanup

The following command will
- Delete all docker images for old containers
- Delete all stopped and orphan containers

Are you sure (Y/n): Y
Starting Cleanup (bytes free 716712)
date: invalid date '"2015-09-28 10:45:1'
scripts/docker-gc: line 69: 1443439271 - : syntax error: operand expected (error token is "- ")
Finished Cleanup (bytes free 716692)

```

It’s still using a ton of space:

```
ubuntu@discourse:/var/discourse$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 6.7G 700M 91% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 996M 12K 996M 1% /dev
tmpfs 201M 356K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user

```

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 28, 2015, 11:24am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/5 "2015-09-28T11:24:14Z")

</div>

I need to start running cleanup from a lightweight image …

what does `docker images` return … do a docker rmi on the old discourse image …

also 7 gigs is really tight considering our images are 1.5gigs (once you factor in bootstrapping and so on)

---

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:25am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/6 "2015-09-28T11:25:39Z")

</div>

```
ubuntu@discourse:/var/discourse$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
local_discourse/app latest 9698d6e190bb 49 minutes ago 1.929 GB
<none> <none> 48086d6c592e 2 days ago 1.928 GB
<none> <none> 537dec28027b 2 days ago 1.928 GB
<none> <none> aa2b4af001ab 3 days ago 1.928 GB
samsaffron/discourse 1.0.13 27f52292c186 6 days ago 1.238 GB

```

Why are all those images there?!

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 28, 2015, 11:27am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/7 "2015-09-28T11:27:20Z")

</div>

> [@avorio](#):
>
> 48086d6c592e  
> 537dec28027b  
> aa2b4af001ab

do a docker rmi on those images.

---

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:30am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/8 "2015-09-28T11:30:49Z")

</div>

Done. All deleted. Here’s the disk space situation:

```
ubuntu@discourse:/var/discourse$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 4.5G 2.9G 61% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 996M 12K 996M 1% /dev
tmpfs 201M 356K 200M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1001M 0 1001M 0% /run/shm
none 100M 0 100M 0% /run/user

```

Question is: why did I have 5 images?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [September 28, 2015, 11:33am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/9 "2015-09-28T11:33:25Z")

</div>

My guess… somehow these failed builds are leaving behind an image, we should investigate that @mpalmer at some point.

Also another possibility is that you are running an old version of bash and launcher is relying on features you do not have.

---

<div class="post-metadata">

### Author: ![avorio](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/avorio/32/120939_2.png) [@avorio](https://meta.discourse.org/u/avorio)
#### Post date: [September 28, 2015, 11:48am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/10 "2015-09-28T11:48:26Z")

</div>

> [@sam](#):
>
> My guess… somehow these failed builds are leaving behind an image, we should investigate that @mpalmer at some point.

But the first failure was caused by lack of disk space. There was no other cause for failure, which suggests to me that your assumption is not correct. Are we missing something else here?

> [@sam](#):
>
> Also another possibility is that you are running an old version of bash and launcher is relying on features you do not have.

Would you like me to provide more info on their versions? Let me know how and I’ll happily do that.

Thanks again, @sam, for your help. The forum is back up now!

---

<div class="post-metadata">

### Author: ![Sanzhar\_Aubakirov](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sanzhar_aubakirov/32/121874_2.png) [@Sanzhar\_Aubakirov](https://meta.discourse.org/u/Sanzhar_Aubakirov)
#### Post date: [April 4, 2016, 5:08am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/11 "2016-04-04T05:08:55Z")

</div>

This is still a problem.

Today I have same `Errno::ENOSPC: No space left on device`

I was rebuilding app a few times before this error occur. `./launcher cleanup` helped

---

<div class="post-metadata">

### Author: ![mkdlin](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@mkdlin](https://meta.discourse.org/u/mkdlin)
#### Post date: [December 21, 2016, 4:25am UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/12 "2016-12-21T04:25:11Z")

</div>

@sam I’m trying to remove an existing image but I can’t seem to remove it

```
root@ip-172-31-27-60:/var/discourse# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
local_discourse/app latest 2bc4af777b30 12 days ago 2.284 GB
discourse/discourse 1.3.8 7f8853cc1cb9 4 weeks ago 1.534 GB

root@ip-172-31-27-60:/var/discourse# docker rmi 7f8853cc1cb9
Error response from daemon: conflict: unable to delete 7f8853cc1cb9 (cannot be forced) - image has dependent child images

```

I’m not sure how else to free up space on my system (I’m not exactly a tech guru)

```
root@ip-172-31-27-60:/var/discourse# df -h
Filesystem Size Used Avail Use% Mounted on
udev 476M 0 476M 0% /dev
tmpfs 100M 11M 89M 11% /run
/dev/xvda1 7.8G 7.1G 253M 97% /
tmpfs 496M 1.3M 495M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
none 7.8G 7.1G 253M 97% /var/lib/docker/aufs/mnt/7a2649494208457846b47f6209a368e3986d8b2bfaf3ff9b2dcb25b14f3ece0d
shm 64M 4.0K 64M 1% /var/lib/docker/containers/90f51aff6b681c0b16084c37c3934d117f006dbdbeb9465ab87fa97db6d1983b/shm
tmpfs 100M 0 100M 0% /run/user/1001

```

Any suggestions?

I have very limited free space on discourse right now and I’m afraid it may crash or something.  
I understand my total space is quite small but that’s because I’m hosting on AWS (t2.micro)

Also I get this error when I try to run cleanup:

```
root@ip-172-31-27-60:/var/discourse# sudo ./launcher cleanup

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image 'discourse/discourse:1.3.9' locally
1.3.9: Pulling from discourse/discourse
b87f06441b40: Pulling fs layer
69c598d5b6ca: Pulling fs layer
b87f06441b40: Verifying Checksum
b87f06441b40: Download complete
69c598d5b6ca: Verifying Checksum
69c598d5b6ca: Download complete
/usr/bin/docker: failed to register layer: Error processing tar file(exit status 1): write /var/lib/dpkg/status-old: no space left on device.
See '/usr/bin/docker run --help'.
Your Docker installation is not working correctly

```

See: [Troubleshoot docker installation issues](https://meta.discourse.org/t/docker-error-on-bootstrap/13657/18)

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [December 21, 2016, 1:05pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/13 "2016-12-21T13:05:24Z")

</div>

Do a backup and then:

> [@Docker installation not working](https://meta.discourse.org/t/docker-installation-not-working/52448/12):
>
> docker stop $(docker ps -a -q)  
> docker rm $(docker ps -a -q)  
> docker rmi $(docker images -q)  
> ./launcher bootstrap app

---

<div class="post-metadata">

### Author: ![mkdlin](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@mkdlin](https://meta.discourse.org/u/mkdlin)
#### Post date: [December 21, 2016, 3:51pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/14 "2016-12-21T15:51:27Z")

</div>

I wanted to login to the admin UI from browser just now but I think it may be broken due to the lack of space?  
(The whole layout of the page is suddenly all wrong)

Is this the guide I want to follow? I already have a good amount of data inside the discourse and I don’t want to mess it up.

> [@Manually create and restore Discourse backups](https://meta.discourse.org/t/advanced-manual-method-of-manually-creating-and-restoring-discourse-backups/18273):
>
> As discussed [below](https://meta.discourse.org/t/advanced-manual-method-of-manually-creating-and-restoring-discourse-backups/18273/7), a warning for newbies: this is an advanced method! there is an automated, web UI way to restore from backups at /admin/backups which is so much easier. with the method below, no automatic backup is kept of the /var/discourse/containers/app.yml file which contains essential SMTP credentials and SSL customizations. Keep a copy of this offsite whenever you change it. If you are using digital ocean, keep frequent snapshots of your droplets - preferably before each time you run .…

---

<div class="post-metadata">

### Author: ![mkdlin](https://avatars.discourse-cdn.com/v4/letter/m/49beb7/32.png) [@mkdlin](https://meta.discourse.org/u/mkdlin)
#### Post date: [December 21, 2016, 5:19pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/15 "2016-12-21T17:19:35Z")

</div>

Managed to fix the issue

- expanded to a larger volume on AWS, which allowed me to do cleanup and also rebuild the app

Initially got a 502 bad gateway but that was also fixed after I rebuilt the app  
Seems to be working normally now, thanks @Falco!

---

<div class="post-metadata">

### Author: ![xiasummer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/xiasummer/32/82790_2.png) [@xiasummer](https://meta.discourse.org/u/xiasummer)
#### Post date: [June 1, 2017, 5:29pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/16 "2017-06-01T17:29:42Z")

</div>

Hi, I encounter the same problem. So you mean I should ask the KVM company to provide more space first?

Do you have anyother method to solve the “No space left” problem?

Yours

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [June 1, 2017, 5:33pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/17 "2017-06-01T17:33:14Z")

</div>

If you are still low on space after a successful cleanup you need to buy more space.

---

<div class="post-metadata">

### Author: ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [June 8, 2024, 12:44pm UTC](https://meta.discourse.org/t/errno-enospc-no-space-left-on-device/33829/18 "2024-06-08T12:44:38Z")

</div>

This topic was automatically closed after 3176 days. New replies are no longer allowed.