Rebuild Failed, please help!

Installation
1

I did a rebuild tonight, and it failed, I disabled all plugins, and it is still failing, unfortunately. It appears some of the gems may not be updating properly. @pfaffman I may need your assistance on this.

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --deployment --retry 3 --jobs 4 --verbose --without test development' failed with return #<Process::Status: pid 367 exit 5>
Location of failure: /usr/local/lib/ruby/gems/2.7.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --retry 3 --jobs 4 --verbose --without test development'"]}
bootstrap failed with exit code 5
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

I, [2022-09-26T00:37:54.244078 #1]  INFO -- : Terminating async processes
I, [2022-09-26T00:37:54.244262 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
2022-09-26 00:37:54.244 UTC [41] LOG:  received fast shutdown request
I, [2022-09-26T00:37:54.244884 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
102:signal-handler (1664152674) Received SIGTERM scheduling shutdown...
2022-09-26 00:37:54.247 UTC [41] LOG:  aborting any active transactions
2022-09-26 00:37:54.250 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2022-09-26 00:37:54.253 UTC [45] LOG:  shutting down
102:M 26 Sep 2022 00:37:54.271 # User requested shutdown...
102:M 26 Sep 2022 00:37:54.271 * Saving the final RDB snapshot before exiting.
2022-09-26 00:37:54.274 UTC [41] LOG:  database system is shut down
102:M 26 Sep 2022 00:37:55.122 * DB saved on disk
102:M 26 Sep 2022 00:37:55.122 # Redis is now ready to exit, bye bye...

Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/msgpack-1.5.6.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/image_size-3.1.0.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/logster-2.11.3.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/version_gem-1.1.0.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/msgpack-1.5.6.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/image_size-3.1.0.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/logster-2.11.3.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/stackprof-0.2.21.gem)
Retrying download gem from https://rubygems.org/ due to error (4/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/image_size-3.1.0.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/omniauth-1.9.2.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/oauth-tty-1.0.3.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/stackprof-0.2.21.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/loofah-2.19.0.gem)
Retrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/css_parser-1.12.0.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/oauth-tty-1.0.3.gem)
Retrying download gem from https://rubygems.org/ due to error (4/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/stackprof-0.2.21.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/loofah-2.19.0.gem)
Retrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/css_parser-1.12.0.gem)
Retrying download gem from https://rubygems.org/ due to error (4/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/oauth-tty-1.0.3.gem)
Retrying download gem from https://rubygems.org/ due to error (4/4): Gem::RemoteFetcher::FetchError Net::OpenTimeout: execution expired (https://rubygems.org/gems/loofah-2.19.0.gem)
Bundler::InstallError: Bundler::HTTPError: Could not download gem from https://rubygems.org/ due to underlying error <Net::OpenTimeout: execution expired (https://rubygems.org/gems/oauth-tty-1.0.3.gem)>
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/rubygems_integration.rb:497:in `rescue in download_gem'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/rubygems_integration.rb:469:in `download_gem'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/source/rubygems.rb:539:in `download_gem'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/source/rubygems.rb:488:in `fetch_gem'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/source/rubygems.rb:465:in `fetch_gem_if_possible'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/source/rubygems.rb:158:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/worker.rb:62:in `apply_func'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/worker.rb:54:in `loop'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/worker.rb:54:in `process_queue'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.18/lib/bundler/worker.rb:91:in `block (2 levels) in create_threads'
2

If requests to rubygems are timing out then that’s likely a problem at their end.

Have you successfully restarted the existing container?

I’m assuming other DNS requests from the server are succeeding?

2 Likes
3

No, I have not been able to start the container

Here’s my output from gem env fyi

RubyGems Environment:
  - RUBYGEMS VERSION: 2.5.2.1
  - RUBY VERSION: 2.3.1 (2016-04-26 patchlevel 112) [x86_64-linux-gnu]
  - INSTALLATION DIRECTORY: /var/lib/gems/2.3.0
  - USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.3.0
  - RUBY EXECUTABLE: /usr/bin/ruby2.3
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - SPEC CACHE DIRECTORY: /root/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /var/lib/gems/2.3.0
     - /root/.gem/ruby/2.3.0
     - /usr/lib/x86_64-linux-gnu/rubygems-integration/2.3.0
     - /usr/share/rubygems-integration/2.3.0
     - /usr/share/rubygems-integration/all
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/games
     - /usr/local/games
     - /snap/bin

And it seems to be connecting:

root@community:/var/discourse# curl -O https://rubygems.org/gems/msgpack-1.5.6.gem
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 89600  100 89600    0     0   745k      0 --:--:-- --:--:-- --:--:--  741k
4

EDIT: Wrong. Well, this was bizarre. It turns out that if you dont’ follow the instructions in Discourse FontAwesome Pro and clone the plugin with the others instead of in a after_code stanza, you get these errors. I can’t explain why it worked before. I looked back through the instructions there to see if this was a change, but it hadn’t been changed recently.

2 Likes
5

I don’t understand what you’re saying Jay, the normal way of installing a plugin is to clone it in an after_code block?

This works for me, I am curious what the bad configuration looks like?

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-fontawesome-pro.git
2 Likes
6

Now I’m really confused. I never noticed which block the plugins are normally in. But I swear it didn’t work with the plugin with all of the others and then I added another after_code block with the git clone and the install script.

I’ll see if I can break it again. :confused: :crying_cat_face:

2 Likes
7

Well, best I can tell, it was just rubygems rate limiting. I’m not quite sure how the brand new droplet I spun up was rate limited, but that, or a spurious issue with rubygems itself, is my best explanation.

2 Likes
8

Try adding 127.0.0.1 to host dns.
That worked for me for this error.