Discourse down after failed web update, rebuild & doctor not helping

I ran a web update today of Discourse + all plugins, and midway in the update there was a message that the update failed. The forum also went down with a message saying something like “the software powering this forum” is having an issue.

I tried git pull and rebuild, but it failed with the following message and left the forum completely offline:

Done 'chunk.178.0c434e9ed79f6e7671b2-ef171b1612b472ed8d08fd8dc9f8a4e4b100b9e910c9285348e11a55846a9823.js' : 0.01 secs

Done 'Done compressing all JS files' : 26.75 secs

rake aborted!
NameError: undefined local variable or method `name' for main:Object
/var/www/discourse/lib/tasks/assets.rake:288:in `rescue in block (2 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:280:in `block (2 levels) in <main>'

Caused by:
OpenURI::HTTPError: 401 Error
/var/www/discourse/lib/file_helper.rb:77:in `block in download'
/var/www/discourse/lib/final_destination.rb:494:in `block (3 levels) in safe_get'
/var/www/discourse/lib/final_destination.rb:494:in `catch'
/var/www/discourse/lib/final_destination.rb:494:in `block (2 levels) in safe_get'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.0.0/lib/patches/net_patches.rb:19:in `block in request_with_mini_profiler'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.0.0/lib/mini_profiler/profiling_methods.rb:46:in `step'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.0.0/lib/patches/net_patches.rb:18:in `request_with_mini_profiler'
/var/www/discourse/lib/final_destination.rb:455:in `block in safe_get'
/var/www/discourse/lib/final_destination.rb:518:in `block in safe_session'
/var/www/discourse/lib/final_destination.rb:511:in `safe_session'
/var/www/discourse/lib/final_destination.rb:450:in `safe_get'
/var/www/discourse/lib/final_destination.rb:161:in `get'
/var/www/discourse/lib/file_helper.rb:69:in `download'
/var/www/discourse/lib/discourse_ip_info.rb:40:in `mmdb_download'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (3 levels) in <main>'
/var/www/discourse/lib/tasks/assets.rake:281:in `each'
/var/www/discourse/lib/tasks/assets.rake:281:in `block (2 levels) in <main>'
Tasks: TOP => assets:precompile
(See full trace by running task with --trace)
I, [2023-03-16T17:44:14.987045 #1]  INFO -- : Checking 'Auto linkify words' for 'default'... up to date
Checking 'Custom Header Links' for 'default'... up to date
Checking 'Guest Gate Theme Component' for 'default'... up to date
Checking 'Guest Gate Theme Component' for 'default'... up to date
yarn run v1.22.19
$ /var/www/discourse/app/assets/javascripts/node_modules/.bin/ember build -prod
WARNING: Node v18.15.0 is not tested against Ember CLI on your platform. We recommend that you use the most-recent "Active LTS" version of Node.js. See https://git.io/v7S5n for details.
Environment: production
cleaning up...
Built project successfully. Stored in "dist/".
File sizes:
 - dist/assets/activate-account.js: 641 B (323 B gzipped)
 - dist/assets/admin.js: 942.87 KB (143.62 KB gzipped)
 - dist/assets/auto-redirect.js: 207 B (173 B gzipped)
 - dist/assets/browser-detect.js: 1001 B (579 B gzipped)
 - dist/assets/browser-update.js: 3.06 KB (1.4 KB gzipped)
 - dist/assets/chunk.13.4e7138edf04d01853515.js: 325.34 KB (99.81 KB gzipped)
 - dist/assets/chunk.143.866024f2070944c8068d.js: 3.83 KB (1.36 KB gzipped)
 - dist/assets/chunk.178.0c434e9ed79f6e7671b2.js: 2.06 KB (1.01 KB gzipped)
 - dist/assets/chunk.682.76d5109d950484255de3.js: 834.41 KB (282.19 KB gzipped)
 - dist/assets/core-tests.js: 2.5 MB (323.5 KB gzipped)
 - dist/assets/discourse.js: 3.1 MB (623.63 KB gzipped)
 - dist/assets/embed-application.js: 1.87 KB (757 B gzipped)
 - dist/assets/google-tag-manager.js: 1.01 KB (558 B gzipped)
 - dist/assets/google-universal-analytics-v3.js: 942 B (499 B gzipped)
 - dist/assets/google-universal-analytics-v4.js: 702 B (349 B gzipped)
 - dist/assets/highlightjs/highlight-test-bundle.min.js: 133.83 KB (42.86 KB gzipped)
 - dist/assets/markdown-it-bundle.js: 129.78 KB (43.63 KB gzipped)
 - dist/assets/onpopstate-handler.js: 265 B (206 B gzipped)
 - dist/assets/plugins/chat.js: 576.67 KB (98.03 KB gzipped)
 - dist/assets/plugins/discourse-akismet.js: 6.42 KB (1.36 KB gzipped)
 - dist/assets/plugins/discourse-chat-integration.js: 46.56 KB (7.04 KB gzipped)
 - dist/assets/plugins/discourse-custom-wizard.js: 314.63 KB (51.01 KB gzipped)
 - dist/assets/plugins/discourse-data-explorer.js: 87.15 KB (16.19 KB gzipped)
 - dist/assets/plugins/discourse-details.js: 1.32 KB (674 B gzipped)
 - dist/assets/plugins/discourse-follow.js: 25.79 KB (4.72 KB gzipped)
 - dist/assets/plugins/discourse-guest-gate.js: 10.49 KB (3.77 KB gzipped)
 - dist/assets/plugins/discourse-local-dates.js: 34.7 KB (9.34 KB gzipped)
 - dist/assets/plugins/discourse-locations.js: 75.06 KB (16.25 KB gzipped)
 - dist/assets/plugins/discourse-narrative-bot.js: 1.66 KB (791 B gzipped)
 - dist/assets/plugins/discourse-news.js: 17.15 KB (4.26 KB gzipped)
 - dist/assets/plugins/discourse-presence.js: 10.95 KB (2.63 KB gzipped)
 - dist/assets/plugins/discourse-reactions.js: 43.75 KB (9.82 KB gzipped)
 - dist/assets/plugins/discourse-telegram-notifications.js: 1.15 KB (512 B gzipped)
 - dist/assets/plugins/docker_manager_admin.js: 36.42 KB (6.96 KB gzipped)
 - dist/assets/plugins/lazy-yt.js: 4.12 KB (1.89 KB gzipped)
 - dist/assets/plugins/poll.js: 56 KB (15.31 KB gzipped)
 - dist/assets/plugins/styleguide.js: 93.27 KB (12.69 KB gzipped)
 - dist/assets/plugins/test/chat_tests.js: 174.2 KB (22.76 KB gzipped)
 - dist/assets/plugins/test/discourse-chat-integration_tests.js: 4.94 KB (1.16 KB gzipped)
 - dist/assets/plugins/test/discourse-custom-wizard_tests.js: 44.24 KB (9.33 KB gzipped)
 - dist/assets/plugins/test/discourse-data-explorer_tests.js: 24.95 KB (3.77 KB gzipped)
 - dist/assets/plugins/test/discourse-details_tests.js: 3.94 KB (1.13 KB gzipped)
 - dist/assets/plugins/test/discourse-follow_tests.js: 4.7 KB (1.76 KB gzipped)
 - dist/assets/plugins/test/discourse-local-dates_tests.js: 28.71 KB (5.54 KB gzipped)
 - dist/assets/plugins/test/discourse-locations_tests.js: 75.97 KB (16.48 KB gzipped)
 - dist/assets/plugins/test/discourse-presence_tests.js: 5.55 KB (1.24 KB gzipped)
 - dist/assets/plugins/test/discourse-reactions_tests.js: 23.95 KB (4.5 KB gzipped)
 - dist/assets/plugins/test/docker_manager_tests.js: 6.16 KB (1.74 KB gzipped)
 - dist/assets/plugins/test/poll_tests.js: 58.71 KB (8.42 KB gzipped)
 - dist/assets/print-page.js: 118 B (130 B gzipped)
 - dist/assets/scripts/discourse-boot.js: 449 B (312 B gzipped)
 - dist/assets/scripts/discourse-test-listen-boot.js: 99 B (101 B gzipped)
 - dist/assets/scripts/discourse-test-load-dynamic-js.js: 637 B (400 B gzipped)
 - dist/assets/scripts/discourse-test-trigger-ember-cli-boot.js: 110 B (104 B gzipped)
 - dist/assets/scripts/module-shims.js: 944 B (471 B gzipped)
 - dist/assets/scripts/start-app.js: 201 B (174 B gzipped)
 - dist/assets/splash-screen.js: 1.8 KB (690 B gzipped)
 - dist/assets/start-discourse.js: 798 B (462 B gzipped)
 - dist/assets/test-helpers.js: 1.02 MB (149.21 KB gzipped)
 - dist/assets/test-i18n.js: 508.39 KB (119.66 KB gzipped)
 - dist/assets/test-site-settings.js: 9.46 KB (3.67 KB gzipped)
 - dist/assets/vendor.css: 0 B
 - dist/assets/vendor.js: 1.11 MB (302.24 KB gzipped)
 - dist/assets/wizard.js: 113.27 KB (34.56 KB gzipped)
 - dist/javascripts/@json-editor/json-editor/2.6.1/jsoneditor.js: 425.11 KB (79.36 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/ace.js: 363.09 KB (98.66 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/ext-searchbox.js: 11.63 KB (3.49 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/mode-html.js: 59.46 KB (17.82 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/mode-scss.js: 23.47 KB (7.79 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/mode-sql.js: 4.73 KB (1.82 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/mode-yaml.js: 4.97 KB (1.73 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/theme-chaos.js: 3.11 KB (1.01 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/theme-chrome.js: 2.97 KB (1.03 KB gzipped)
 - dist/javascripts/ace.js/1.4.13/worker-html.js: 203.22 KB (46.33 KB gzipped)
 - dist/javascripts/ace/ace.js: 362.06 KB (98.41 KB gzipped)
 - dist/javascripts/ace/ext-searchbox.js: 11.63 KB (3.49 KB gzipped)
 - dist/javascripts/ace/mode-html.js: 59.46 KB (17.82 KB gzipped)
 - dist/javascripts/ace/mode-scss.js: 23.47 KB (7.79 KB gzipped)
 - dist/javascripts/ace/mode-sql.js: 2.16 KB (1009 B gzipped)
 - dist/javascripts/ace/mode-yaml.js: 4.97 KB (1.73 KB gzipped)
 - dist/javascripts/ace/theme-chrome.js: 2.97 KB (1.03 KB gzipped)
 - dist/javascripts/ace/worker-html.js: 203.22 KB (46.33 KB gzipped)
 - dist/javascripts/chart.js/3.5.1/chart.min.js: 183.95 KB (63.67 KB gzipped)
 - dist/javascripts/chartjs-plugin-datalabels/2.0.0/chartjs-plugin-datalabels.min.js: 12.59 KB (5.09 KB gzipped)
 - dist/javascripts/count.js: 1.41 KB (636 B gzipped)
 - dist/javascripts/diffhtml/1.0.0-beta.20/diffhtml.min.js: 26.98 KB (9.88 KB gzipped)
 - dist/javascripts/embed-topics.js: 1.31 KB (630 B gzipped)
 - dist/javascripts/embed.js: 2.41 KB (1010 B gzipped)
 - dist/javascripts/magnific-popup/1.1.0/jquery.magnific-popup.min.js: 19.74 KB (7.18 KB gzipped)
 - dist/javascripts/media-optimization-worker.js: 4.89 KB (1.71 KB gzipped)
 - dist/javascripts/pikaday/1.8.2/pikaday.js: 41.55 KB (9.44 KB gzipped)
 - dist/javascripts/squoosh/mozjpeg_enc.js: 38.12 KB (10.49 KB gzipped)
 - dist/javascripts/squoosh/squoosh_resize.js: 4.42 KB (1.26 KB gzipped)
 - dist/javascripts/workbox/workbox-cacheable-response.dev.js: 6.58 KB (1.73 KB gzipped)
 - dist/javascripts/workbox/workbox-cacheable-response.prod.js: 579 B (345 B gzipped)
 - dist/javascripts/workbox/workbox-core.dev.js: 48.27 KB (11.21 KB gzipped)
 - dist/javascripts/workbox/workbox-core.prod.js: 5.75 KB (2.42 KB gzipped)
 - dist/javascripts/workbox/workbox-expiration.dev.js: 20.83 KB (5.48 KB gzipped)
 - dist/javascripts/workbox/workbox-expiration.prod.js: 2.82 KB (1.22 KB gzipped)
 - dist/javascripts/workbox/workbox-routing.dev.js: 31.54 KB (7.58 KB gzipped)
 - dist/javascripts/workbox/workbox-routing.prod.js: 3.32 KB (1.44 KB gzipped)
 - dist/javascripts/workbox/workbox-strategies.dev.js: 34.65 KB (4.79 KB gzipped)
 - dist/javascripts/workbox/workbox-strategies.prod.js: 4.75 KB (1.16 KB gzipped)
 - dist/javascripts/workbox/workbox-sw.js: 1.3 KB (741 B gzipped)
Done in 222.00s.
Downloading MaxMindDB...
Compressing Javascript and Generating Source Maps

I, [2023-03-16T17:44:14.990316 #1]  INFO -- : Terminating async processes
I, [2023-03-16T17:44:14.990705 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
I, [2023-03-16T17:44:14.990950 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 102
2023-03-16 17:44:14.991 UTC [41] LOG:  received fast shutdown request
102:signal-handler (1678988654) Received SIGTERM scheduling shutdown...
2023-03-16 17:44:14.995 UTC [41] LOG:  aborting any active transactions
2023-03-16 17:44:15.002 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
102:M 16 Mar 2023 17:44:15.053 # User requested shutdown...
102:M 16 Mar 2023 17:44:15.054 * Saving the final RDB snapshot before exiting.
2023-03-16 17:44:15.184 UTC [45] LOG:  shutting down
2023-03-16 17:44:15.252 UTC [41] LOG:  database system is shut down
102:M 16 Mar 2023 17:44:16.440 * DB saved on disk
102:M 16 Mar 2023 17:44:16.441 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 390 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Any idea what can be causing it?

I also ran discourse doctor, which ended up creating 502 errors. I commented out any third-party plugins I have installed and tried rebuilding again, but no dice…

Not really sure what my next step would be here… Any ideas?

You might want to comment out your MaxMindDB API key, it seems that is what is failing.

3 Likes

@RGJ amazing, thank you! You just saved me from nearly having a heart attack. I do now recall adding in the API key some time ago so that hopefully I would get the MaxMind functionality working for the forum whenever I did the next rebuild. Guess that came back to bite me!

Any idea why the MaxMind key is causing the site to fail building? Could it be an incorrect API key or something else? I had entered it in the following format:

DISCOURSE_MAXMIND_LICENSE_KEY: kEyHeR3

Should it be entered with surrounding "s?

Edit: The key is showing as being “used” just now, so I think I’ve entered the correct key:

3 Likes

Hey @Falco, here’s another one of these failing because of maxmind.

2 Likes

@Falco if it would help you reproduce I’d be happy to share my specific MaxMind key in a PM.

1 Like

Is your maxmind key valid?

Thanks, but no need, since as far as I can see any invalid key or gibberish will trigger the bug.

1 Like

It doesn’t seem to be invalid, though… See my screenshot above, it even registers use for the key during the rebuild… :man_shrugging:

It was generated from my MaxMind account on https://www.maxmind.com/en/accounts/[MyAccountID]/license-key.

Anyway, I’ll try with a new key and see if it works now.

EDIT: Nope, that didn’t work. Generated a new key, added it to app.yml, rebuilt. Still failed in the same way.

I also verified that my account has access to GeoLite 2:

I also accepted their GeoLite 2 EULA again for good measure.

I’m a paid customer of their GeoIP2 Insights service, not sure if that could contribute to the issues?

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.