Stuck in a loop of freeing up space and filling it up again when rebuilding

How much disk space do you have?

Can you delete some backups?

1 Like

Sure…

Have you ran ./launcher cleanup it will purge all images not in use.

1 Like
$ df -h
Filesystem              Size  Used Avail Use% Mounted on
udev                    476M     0  476M   0% /dev
tmpfs                   100M  8.9M   91M   9% /run
/dev/mapper/vg-lv_root   19G   14G  4.1G  77% /
tmpfs                   497M  1.1M  496M   1% /dev/shm
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                   497M     0  497M   0% /sys/fs/cgroup
/dev/sda1               461M  160M  278M  37% /boot
tmpfs                   100M     0  100M   0% /run/user/1000

Deleted them already.

Yes, I tried that. and I get

But when I try to rebuild or start the app, the base images that was presumably deleted by cleanup gets downloaded again and I’m back to where I started.

My suggestion here then if you are ULTRA tight on space is just to go nuclear and and start from scratch.

What is the output of docker ps, you basically need to kill your app container so you can free up the old base image.

$ docker ps
CONTAINER ID        IMAGE                           COMMAND             CREATED             STATUS              PORTS                                      NAMES
1fba0860cbc3        local_discourse/web_only        "/sbin/boot"        5 months ago        Up 29 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   web_only
aa6b422d88ca        local_discourse/data            "/sbin/boot"        8 months ago        Up 29 minutes                                                  data
2940a1603151        local_discourse/mail-receiver   "/sbin/boot"        8 months ago        Up 29 minutes       0.0.0.0:25->25/tcp                         mail-receiver

What do you mean by that? Will I need a discourse backup? Cause I don’t have one…

docker rm -f aa6b422d88ca
docker rm -f 1fba0860cbc3
./launcher cleanup 
./launcher rebuild data
./launcher rebuild web_only

But really the root cause here is that you are just way too tight on disk space. I would recommend doubling the size /dev/mappper/vg-lv_root

3 Likes

The last step (rebuild web_only) fails:

Bundle complete! 110 Gemfile dependencies, 201 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2018-08-31T00:50:57.518677 #13]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::LoadError: can't activate public_suffix-2.0.5, already activated public_suffix-3.0.2
/var/www/discourse/lib/plugin_gem.rb:18:in `load'
/var/www/discourse/lib/plugin/instance.rb:501:in `gem'
/var/www/discourse/plugins/discourse-sync-to-dropbox/plugin.rb:7:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:431:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:431:in `activate!'
lib/discourse.rb:164:in `block in activate_plugins!'
lib/discourse.rb:161:in `each'
lib/discourse.rb:161:in `activate_plugins!'
/var/www/discourse/config/application.rb:223:in `<class:Application>'
/var/www/discourse/config/application.rb:39:in `<module:Discourse>'
/var/www/discourse/config/application.rb:38:in `<top (required)>'
/var/www/discourse/Rakefile:5:in `require'
/var/www/discourse/Rakefile:5:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
I, [2018-08-31T00:51:18.343389 #13]  INFO -- : gem install geocoder -v 1.4.4 -i /var/www/discourse/plugins/discourse-locations/gems/2.5.1 --no-document --ignore-dependencies
Successfully installed geocoder-1.4.4
1 gem installed
gem install public_suffix -v 2.0.5 -i /var/www/discourse/plugins/discourse-sync-to-dropbox/gems/2.5.1 --no-document --ignore-dependencies
Successfully installed public_suffix-2.0.5
1 gem installed




FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 507 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
36b568dc13935b0491b99afd4763776899dad8d57a1df149633b9c7c02fd3c0b
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

Trying without the dropbox plugin now…

Hmmmm … are you on tests passed… stable or beta?

Also, remove discourse-sync-to-dropbox asap. It looks like it is not working at the moment.

3 Likes

Phew! After removing the dropbox plugin my site is finally back up and running. Thanks a lot to everyone for your support!

Since I never changed that, I guess I’m on tests-passed.

I guess you’re right, but how does that relate to discourse’s minimum requirements of 10 GB HDD?

10GB is ambitious and absolutely not the case if you are running seperate data and web containers.

4 Likes

Ah, I see. I don’t think that was mentioned anywhere. I’ll check tomorrow where that information might be usefully added.

However, the situation seems to look much brighter now:

$ df -h
Filesystem              Size  Used Avail Use% Mounted on
udev                    476M     0  476M   0% /dev
tmpfs                   100M   12M   88M  12% /run
/dev/mapper/vg-lv_root   19G  8.4G  9.0G  49% /
tmpfs                   497M  1.4M  495M   1% /dev/shm
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                   497M     0  497M   0% /sys/fs/cgroup
/dev/sda1               461M  160M  278M  37% /boot
tmpfs                   100M     0  100M   0% /run/user/1000

Not sure where all the space came from but it looks good, doesn’t it? Do you still think I need to upgrade the server?

The data container was using the old image, so you had a situation where:

“data container” was saying HEY I need the old Discourse base image
“web container” was saying I need the new image

Not sure where all the space came from but this is a chunk of it.

1 Like

How did that happen?

From my side, I can say that I started an update via /admin/upgrade which failed, presumably due to a lack of space. Since the website was down, I tried to rebuild via ssh and that’s when the whole misery started…

Regardless of this specific situation, I wonder if the web-updater could not handle lack of disk space better. Rather than crashing the site, could it not just exit before it’s too late?

Because you hadn’t upgraded the data container.

It could probably do a disk space check before it started, and refuse to proceed if there wasn’t enough space.

3 Likes

Another question:

Where did you get these IDs from:

Those are the container IDs (first column in the docker ps output).

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

I still don’t quite understand this. If I’m supposed to upgrade the data conatainer too, why doesn’t it appear in these instructions?

And isn’t the whole point of the two container setup that I don’t have to rebuild the data container? Well, I guess, not, but if someone could help me understand what the proper procedure for a two container setup is when /admin/upgrade/ tells me to do

cd /var/discourse/
git pull
./launcher rebuild app

Hence the command you quoted is talking about the web_only container… not the data container.

2 container setup is not something we detect in the web upgrader. The commands you want to run are exactly the ones you quoted…

1 Like