Site wont rebuild (failed to bootstrap) after server update


#1

If someone could please help me on this. I did a server update on digital ocean and now site wont rebuild. Not sure where to go from here

ications/activerecord-5.2.0.gemspec
Fetching aws-sdk-s3 1.14.0
Installing aws-sdk-s3 1.14.0
2:  aws-sdk-s3 (1.14.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/aws-sdk-s3-1.14.0.gemspec
Using omniauth-oauth2 1.5.0
2:  omniauth-oauth2 (1.5.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-oauth2-1.5.0.gemspec
Using omniauth-twitter 1.4.0
2:  omniauth-twitter (1.4.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-twitter-1.4.0.gemspec
Using sass 3.5.6
2:  sass (3.5.6) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/sass-3.5.6.gemspec
Using seed-fu 2.3.9
2:  seed-fu (2.3.9) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/seed-fu-2.3.9.gemspec
Using shoulda 3.5.0
2:  shoulda (3.5.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/shoulda-3.5.0.gemspec
Using omniauth-facebook 5.0.0
2:  omniauth-facebook (5.0.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-facebook-5.0.0.gemspec
Using omniauth-github 1.3.0
2:  omniauth-github (1.3.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-github-1.3.0.gemspec
Using omniauth-google-oauth2 0.5.3
2:  omniauth-google-oauth2 (0.5.3) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-google-oauth2-0.5.3.gemspec
Using omniauth-instagram 1.3.0
2:  omniauth-instagram (1.3.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/omniauth-instagram-1.3.0.gemspec
Using sassc 1.11.4
2:  sassc (1.11.4) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/sassc-1.11.4.gemspec
1:  pg (1.0.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/pg-1.0.0.gemspec
0:  nokogiri (1.8.3) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/nokogiri-1.8.3.gemspec
Using rails-dom-testing 2.0.3
2:  rails-dom-testing (2.0.3) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/rails-dom-testing-2.0.3.gemspec
Using loofah 2.2.2
2:  loofah (2.2.2) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/loofah-2.2.2.gemspec
Using nokogumbo 1.5.0
2:  nokogumbo (1.5.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/nokogumbo-1.5.0.gemspec
Using rspec-html-matchers 0.9.1
2:  rspec-html-matchers (0.9.1) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/rspec-html-matchers-0.9.1.gemspec
Using ruby-readability 0.7.0
2:  ruby-readability (0.7.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/ruby-readability-0.7.0.gemspec
Using rails-html-sanitizer 1.0.4
Using sanitize 4.6.5
0:  rails-html-sanitizer (1.0.4) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/rails-html-sanitizer-1.0.4.gemspec
2:  sanitize (4.6.5) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/sanitize-4.6.5.gemspec
Using actionview 5.2.0
1:  actionview (5.2.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/actionview-5.2.0.gemspec
Fetching onebox 1.8.50
Using actionpack 5.2.0
2:  actionpack (5.2.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/actionpack-5.2.0.gemspec
Using actionmailer 5.2.0
0:  actionmailer (5.2.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/actionmailer-5.2.0.gemspec
Using railties 5.2.0
0:  railties (5.2.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/railties-5.2.0.gemspec
Using sprockets-rails 3.2.1
0:  sprockets-rails (3.2.1) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/sprockets-rails-3.2.1.gemspec
Using jquery-rails 4.3.3
2:  jquery-rails (4.3.3) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/jquery-rails-4.3.3.gemspec
Using lograge 0.10.0
2:  lograge (0.10.0) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/lograge-0.10.0.gemspec
Fetching rails_multisite 2.0.5
Using rspec-rails 3.7.2
0:  rspec-rails (3.7.2) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/rspec-rails-3.7.2.gemspec
Using ember-rails 0.18.5
0:  ember-rails (0.18.5) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/ember-rails-0.18.5.gemspec
Installing rails_multisite 2.0.5
Installing onebox 1.8.50
2:  rails_multisite (2.0.5) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/rails_multisite-2.0.5.gemspec
1:  onebox (1.8.50) from /var/www/discourse/vendor/bundle/ruby/2.5.0/specifications/onebox-1.8.50.gemspec
Bundle complete! 108 Gemfile dependencies, 185 gems now installed.
Gems in the group development were not installed.
Bundled gems are installed into `./vendor/bundle`

I, [2018-06-21T10:43:50.788914 #18]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
rake aborted!
Gem::LoadError: can't activate public_suffix-2.0.5, already activated public_suffix-3.0.2
/var/www/discourse/lib/plugin_gem.rb:18:in `load'
/var/www/discourse/lib/plugin/instance.rb:501:in `gem'
/var/www/discourse/plugins/discourse-sync-to-dropbox/plugin.rb:7:in `activate!'
/var/www/discourse/lib/plugin/instance.rb:446:in `instance_eval'
/var/www/discourse/lib/plugin/instance.rb:446:in `activate!'
/var/www/discourse/lib/discourse.rb:151:in `block in activate_plugins!'
/var/www/discourse/lib/discourse.rb:148:in `each'
/var/www/discourse/lib/discourse.rb:148:in `activate_plugins!'
/var/www/discourse/config/application.rb:218:in `<class:Application>'
/var/www/discourse/config/application.rb:39:in `<module:Discourse>'
/var/www/discourse/config/application.rb:38:in `<top (required)>'
/var/www/discourse/Rakefile:5:in `require'
/var/www/discourse/Rakefile:5:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
I, [2018-06-21T10:44:02.502661 #18]  INFO -- : gem install public_suffix -v 2.0.5 -i /var/www/discourse/plugins/discourse-sync-to-dropbox/gems/2.5.1 --no-document --ignore-dependencies
Successfully installed public_suffix-2.0.5
1 gem installed

I, [2018-06-21T10:44:02.503547 #18]  INFO -- : Terminating async processes
I, [2018-06-21T10:44:02.503822 #18]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/10/bin/postmaster -D /etc/postgresql/10/main pid: 73
2018-06-21 10:44:02.504 UTC [73] LOG:  received fast shutdown request
I, [2018-06-21T10:44:02.504148 #18]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 189
189:signal-handler (1529577842) Received SIGTERM scheduling shutdown...
2018-06-21 10:44:02.505 UTC [73] LOG:  aborting any active transactions
2018-06-21 10:44:02.510 UTC [73] LOG:  worker process: logical replication launcher (PID 82) exited with exit code 1
2018-06-21 10:44:02.510 UTC [77] LOG:  shutting down
2018-06-21 10:44:02.527 UTC [73] LOG:  database system is shut down
189:M 21 Jun 10:44:02.566 # User requested shutdown...
189:M 21 Jun 10:44:02.566 * Saving the final RDB snapshot before exiting.
189:M 21 Jun 10:44:03.035 * DB saved on disk
189:M 21 Jun 10:44:03.035 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 10743 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --verbose --without test --without development --retry 3 --jobs 4'", "su discourse -c 'bundle exec rake db:migrate'", "su discourse -c 'bundle exec rake assets:precompile'"]}
554b092ac832ce1a911e16eb48ed8d143eb414a0c76a11405748c3dd9383c766
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

(Bhanu Sharma) #2

Are you using a customized version of discourse image? Why is a higher number of public_suffix gem already installed?

Have you tried to rebuild?

cd /var/discourse
git pull
./launcher rebuild app

#3

It was a Digital Ocean one click install I believe when it started.

I found an error in my app.yaml file on my test server (I had to fully delete the backup to dropbox) and fixed that. So the test server is working fine.

Now I get a warning docker depreciated on my main server, and only 5gb space left in docker and it needs more space to clear containers. There should be plenty of room on it. I dont have the latest backup of my site and not sure if clearing this will delete the backups too? The ssh download is about 10 hours left on it.


(Bhanu Sharma) #4

Have you tried the following:

Taking a backup from your admin area and downloading it.

Then in terminal:

cd /var/discourse
./launcher cleanup

This should clean up any old/obsolete data on your server which may create some space for you to use.
Alternatively, if you’re willing to pay more, you can either:

Configure Uploads to s3 and run the migrate to s3 rake task which may free up storage space for media/attachments uploaded on your forum and give you some extra space.

Or you can increase the size of your digital ocean droplet to facilitate for the upgrade. However, I don’t think that can be reverted so it will come as an additional cost per month.


#5

No I cant get in admin area since the launcher wont rebuild.

Thats why doing a ssh download of the backup. Just another 10 hours left.

All my problems started when I enabled s3 on my site. After I deleted the keys and info in settings it still kept sending everything to s3. Thats when I went in to rebuild the app


(Bhanu Sharma) #6

Looks like you didn’t took enough precaution.
before stopping uploads to s3 you should’ve really run the task to migrate uploads back to local storage. once that was done, You could’ve disabled the s3 uploads.
but wait! did you had enough storage on local??


#7

I dont even worry about that right now. All I need is the site back up an running. I should have plenty of storage on my droplet.

Now I need to know whether running this clean will delete my backups in the file or not. So I can get this back up faster then waiting 10 hours.


(Bhanu Sharma) #8

As per the launcher documentation on github,

cleanup:    Remove all containers that have stopped for > 24 hours

So it shouldn’t remove anything apart form dead containers.

But I’m not sure, Download the backup Just in case


#9

Said it freed up 17gb but when I go to rebuild it gives same 5gb warning. I remember there was a force command for this on here but cant find it,


(Bhanu Sharma) #10

Can you share a screenshot or copy-paste of what exactly it says?


#11
Digest: sha256:c5cb70244978e0cd95a2e56b3d28668618318e5ef3ffa96ecf3347c1261e2e42
Status: Downloaded newer image for discourse/base:2.0.20180613
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        78G   75G  2.8G  97% /

Would you like to attempt to recover space by cleaning docker images and containers in the system?(y/N)y
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
If the cleanup was successful, you may try again now

#12

now it gives me this when i try cleanup says 0B again and nothing.


(Bhanu Sharma) #13

hmmm! Your primary disk is almost full are you by chance storing contents of something other than discourse to this server?


#14

Nope this is only for discourse.


#15

I went to do autoremove and get this now. To keep the local version?

 A new version of /boot/grub/menu.lst is available, but the version   │
    │ installed currently has been locally modified.                       │
    │                                                                      │
    │ What would you like to do about menu.lst?                            │
    │                                                                      │
    │      install the package maintainer's version                        │
    │      keep the local version currently installed                      │
    │      show the differences between the versions                       │
    │      show a side-by-side difference between the versions             │
    │      show a 3-way difference between available versions              │
    │      do a 3-way merge between available versions (experimental)      │
    │      start a new shell to examine the situation

(Bhanu Sharma) #16

Keep local version is the Known safe choice.


#17

seems to be stuck in a loop everytime i choose that.


(Bhanu Sharma) #18

I’m sure that there would be ways to get this server back online but considering the amount of data that is on there, I’d consider spinning up a second server (with more than 20 GB space) and pulling the latest backup zip from there. This can be done using scp on terminal (use DO private networking)

And then you may be able to manually delete stuff and rebuild?


#19

looks like it may have worked. Kept pressing thru about 20 times and got an error but now its letting me rebuild the app. Keeping fingers crossed lol. I have the second server up as soon as i can get a backup I’ll transfer it all over.


(Bhanu Sharma) #20

If you already have the second server why are you even bothering to rebuild before transferring backup from server to server? why even waste time downloading and re uploading? server link should be multitude of times faster than our internet connection.