Multi-site upgrade leaves _some_ sites unavailable

hellekin · June 11, 2021, 10:12am

With the recent upgrade frenzy, leading to 2.8.0.beta2, something strange happened that I’m currently investigating. The effect is that on a multi-site install, some but not all secondary sites do not respond anymore. The primary site is up, and some secondary sites operate normally.

~~I suspect it may have to do with the discourse-calendar issue already reported on stable, but that’s just a hunch.~~

hellekin · June 11, 2021, 11:21am

I found this in the log for one of the sites:

error log

[thx_zoethical_org] Started GET "/t/nos-reseaux-tisses/77" for [REDACTED IP] at 2021-06-11 11:19:51 +0000
[thx_zoethical_org] Processing by TopicsController#show as HTML
[thx_zoethical_org]   Parameters: {"slug"=>"nos-reseaux-tisses", "topic_id"=>"77"}
[thx_zoethical_org] Completed 500 Internal Server Error in 388ms (ActiveRecord: 0.0ms | Allocations: 74642)
[thx_zoethical_org] NoMethodError (undefined method `include?' for nil:NilClass)
lib/topic_query.rb:111:in `block in apply_custom_filters'
lib/topic_query.rb:110:in `each'
lib/topic_query.rb:110:in `apply_custom_filters'
lib/topic_query.rb:890:in `default_results'
lib/topic_query.rb:523:in `unread_results'
lib/topic_query.rb:218:in `list_suggested_for'
lib/topic_view.rb:572:in `suggested_topics'
app/serializers/suggested_topics_mixin.rb:16:in `include_suggested_topics?'
(eval):6:in `_fast_attributes'
app/controllers/topics_controller.rb:1158:in `block (2 levels) in perform_show_response'
app/controllers/topics_controller.rb:1153:in `perform_show_response'
app/controllers/topics_controller.rb:162:in `show'
app/controllers/application_controller.rb:395:in `block in with_resolved_locale'
app/controllers/application_controller.rb:395:in `with_resolved_locale'
lib/middleware/omniauth_bypass_middleware.rb:71:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:355:in `call'
config/initializers/008-rack-cors.rb:25:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
lib/middleware/enforce_hostname.rb:23:in `call'
lib/middleware/request_tracker.rb:177:in `call'

pfaffman · June 12, 2021, 8:32pm

Did you do a git pull first (unnecessary if you do a rebuild rather then a bootstrap).

Another wild guess is that it’s theme related.

Check themes and plugins. Try safe mode?

hellekin · June 13, 2021, 8:56pm

Thank you @pfaffman. I used rebuild (twice) and safe-mode did not help. I’m currently rebuilding on another machine to avoid multisite and see if restoring from backups work.

Just to add: published pages (under /pub) work normally. Oh, and /admin too… Maybe I can solve it…

Apparently it comes from discourse-prometheus-alert-receiver which is not enabled for this site…

plugins/discourse-prometheus-alert-receiver/plugin.rb:152:in `block (2 levels) in activate!'
lib/topic_query.rb:111:in `block in apply_custom_filters'
lib/topic_query.rb:110:in `each'
lib/topic_query.rb:110:in `apply_custom_filters'
lib/topic_query.rb:890:in `default_results'
lib/topic_query.rb:507:in `latest_results'
lib/topic_query.rb:242:in `list_latest'
app/controllers/categories_controller.rb:51:in `block (2 levels) in index'
actionpack (6.1.3.2) lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
app/controllers/categories_controller.rb:40:in `index'
actionpack (6.1.3.2) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (6.1.3.2) lib/abstract_controller/base.rb:228:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (6.1.3.2) lib/abstract_controller/callbacks.rb:42:in `block in process_action'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:117:in `block in run_callbacks'
app/controllers/application_controller.rb:395:in `block in with_resolved_locale'
i18n (1.8.10) lib/i18n.rb:314:in `with_locale'
app/controllers/application_controller.rb:395:in `with_resolved_locale'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:126:in `block in run_callbacks'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:137:in `run_callbacks'
actionpack (6.1.3.2) lib/abstract_controller/callbacks.rb:41:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
activesupport (6.1.3.2) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.3.2) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.3.2) lib/active_support/notifications.rb:203:in `instrument'
actionpack (6.1.3.2) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
activerecord (6.1.3.2) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (6.1.3.2) lib/abstract_controller/base.rb:165:in `process'
actionview (6.1.3.2) lib/action_view/rendering.rb:39:in `process'
rack-mini-profiler (2.3.2) lib/mini_profiler/profiling_methods.rb:111:in `block in profile_method'


actionpack (6.1.3.2) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.1.3.2) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:842:in `call'
lib/middleware/omniauth_bypass_middleware.rb:71:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
actionpack (6.1.3.2) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:355:in `call'
config/initializers/008-rack-cors.rb:25:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/cookies.rb:689:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:98:in `run_callbacks'
actionpack (6.1.3.2) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
logster (2.9.6) lib/logster/middleware/reporter.rb:43:in `call'
railties (6.1.3.2) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.1.3.2) lib/rails/rack/logger.rb:28:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/request_id.rb:26:in `call'
lib/middleware/enforce_hostname.rb:23:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
plugins/discourse-prometheus/lib/middleware/metrics.rb:17:in `call'
rack-mini-profiler (2.3.2) lib/mini_profiler/profiler.rb:384:in `call'


message_bus (3.3.6) lib/message_bus/rack/middleware.rb:61:in `call'
lib/middleware/request_tracker.rb:177:in `call'
rails_multisite (3.0.0) lib/rails_multisite/middleware.rb:25:in `call'
railties (6.1.3.2) lib/rails/engine.rb:539:in `call'
railties (6.1.3.2) lib/rails/railtie.rb:207:in `public_send'
railties (6.1.3.2) lib/rails/railtie.rb:207:in `method_missing'
rack (2.2.3) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.3) lib/rack/urlmap.rb:58:in `each'
rack (2.2.3) lib/rack/urlmap.rb:58:in `call'
unicorn (6.0.0) lib/unicorn/http_server.rb:634:in `process_client'
unicorn (6.0.0) lib/unicorn/http_server.rb:732:in `worker_loop'
unicorn (6.0.0) lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
unicorn (6.0.0) lib/unicorn/http_server.rb:143:in `start'
unicorn (6.0.0) bin/unicorn:128:in `<top (required)>'
vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `load'
vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `<main>'

The following line seems to be the culprit. But since the plugin is not enabled, it should not fire, right?

github.com

discourse/discourse-prometheus-alert-receiver/blob/main/plugin.rb#L152

    
      
              users.clear
            end
          end
          
          
TopicQuery.add_custom_filter(
            DiscoursePrometheusAlertReceiver::PLUGIN_NAME
          ) do |results, topic_query|
          
          
  options = topic_query.options
          
          
  if Topic.alerts_category_ids_cache.include?(options[:category_id]) && options[:status] == 'firing'
              results = results.firing_alerts([options[:category_id]])
            end
          
          
  results
          end
          
          
require_dependency "admin_constraint"
          
          
::DiscoursePrometheusAlertReceiver::Engine.routes.draw do
            token_format = /[a-f0-9]{64}/

hellekin · June 13, 2021, 9:10pm

When discourse-prometheus-alert-receiver is enabled, the site works fine. When it is not enabled, the site breaks. This sounds like a bug to me.

system · July 13, 2021, 9:11pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Solved Plugin broke site bug solved	7	570	October 18, 2021
Upgrade failure mmdb_download (getaddrinfo: Name or service not known) support	6	1376	March 19, 2020
Upgrading from 1.6.0 to 3 installation	13	509	January 18, 2023
After rebuilding app, can't sign in with error 500 bug chat	35	1137	February 16, 2024
500 Error when loading a specific topic support post-voting	5	481	December 5, 2022

Multi-site upgrade leaves _some_ sites unavailable

Related Topics