Multi-site upgrade leaves _some_ sites unavailable

hellekin · June 11, 2021, 10:12am

With the recent upgrade frenzy, leading to 2.8.0.beta2, something strange happened that I’m currently investigating. The effect is that on a multi-site install, some but not all secondary sites do not respond anymore. The primary site is up, and some secondary sites operate normally.

~~I suspect it may have to do with the discourse-calendar issue already reported on stable, but that’s just a hunch.~~

hellekin · June 11, 2021, 11:21am

I found this in the log for one of the sites:

error log

[thx_zoethical_org] Started GET "/t/nos-reseaux-tisses/77" for [REDACTED IP] at 2021-06-11 11:19:51 +0000
[thx_zoethical_org] Processing by TopicsController#show as HTML
[thx_zoethical_org]   Parameters: {"slug"=>"nos-reseaux-tisses", "topic_id"=>"77"}
[thx_zoethical_org] Completed 500 Internal Server Error in 388ms (ActiveRecord: 0.0ms | Allocations: 74642)
[thx_zoethical_org] NoMethodError (undefined method `include?' for nil:NilClass)
lib/topic_query.rb:111:in `block in apply_custom_filters'
lib/topic_query.rb:110:in `each'
lib/topic_query.rb:110:in `apply_custom_filters'
lib/topic_query.rb:890:in `default_results'
lib/topic_query.rb:523:in `unread_results'
lib/topic_query.rb:218:in `list_suggested_for'
lib/topic_view.rb:572:in `suggested_topics'
app/serializers/suggested_topics_mixin.rb:16:in `include_suggested_topics?'
(eval):6:in `_fast_attributes'
app/controllers/topics_controller.rb:1158:in `block (2 levels) in perform_show_response'
app/controllers/topics_controller.rb:1153:in `perform_show_response'
app/controllers/topics_controller.rb:162:in `show'
app/controllers/application_controller.rb:395:in `block in with_resolved_locale'
app/controllers/application_controller.rb:395:in `with_resolved_locale'
lib/middleware/omniauth_bypass_middleware.rb:71:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:355:in `call'
config/initializers/008-rack-cors.rb:25:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
lib/middleware/enforce_hostname.rb:23:in `call'
lib/middleware/request_tracker.rb:177:in `call'

pfaffman · June 12, 2021, 8:32pm

Did you do a git pull first (unnecessary if you do a rebuild rather then a bootstrap).

Another wild guess is that it’s theme related.

Check themes and plugins. Try safe mode?

hellekin · June 13, 2021, 8:56pm

Thank you @pfaffman. I used rebuild (twice) and safe-mode did not help. I’m currently rebuilding on another machine to avoid multisite and see if restoring from backups work.

Just to add: published pages (under /pub) work normally. Oh, and /admin too… Maybe I can solve it…

Apparently it comes from discourse-prometheus-alert-receiver which is not enabled for this site…

plugins/discourse-prometheus-alert-receiver/plugin.rb:152:in `block (2 levels) in activate!'
lib/topic_query.rb:111:in `block in apply_custom_filters'
lib/topic_query.rb:110:in `each'
lib/topic_query.rb:110:in `apply_custom_filters'
lib/topic_query.rb:890:in `default_results'
lib/topic_query.rb:507:in `latest_results'
lib/topic_query.rb:242:in `list_latest'
app/controllers/categories_controller.rb:51:in `block (2 levels) in index'
actionpack (6.1.3.2) lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
app/controllers/categories_controller.rb:40:in `index'
actionpack (6.1.3.2) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (6.1.3.2) lib/abstract_controller/base.rb:228:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (6.1.3.2) lib/abstract_controller/callbacks.rb:42:in `block in process_action'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:117:in `block in run_callbacks'
app/controllers/application_controller.rb:395:in `block in with_resolved_locale'
i18n (1.8.10) lib/i18n.rb:314:in `with_locale'
app/controllers/application_controller.rb:395:in `with_resolved_locale'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:126:in `block in run_callbacks'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:137:in `run_callbacks'
actionpack (6.1.3.2) lib/abstract_controller/callbacks.rb:41:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
activesupport (6.1.3.2) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.3.2) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.3.2) lib/active_support/notifications.rb:203:in `instrument'
actionpack (6.1.3.2) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
actionpack (6.1.3.2) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
activerecord (6.1.3.2) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (6.1.3.2) lib/abstract_controller/base.rb:165:in `process'
actionview (6.1.3.2) lib/action_view/rendering.rb:39:in `process'
rack-mini-profiler (2.3.2) lib/mini_profiler/profiling_methods.rb:111:in `block in profile_method'


actionpack (6.1.3.2) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.1.3.2) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.1.3.2) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.1.3.2) lib/action_dispatch/routing/route_set.rb:842:in `call'
lib/middleware/omniauth_bypass_middleware.rb:71:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
actionpack (6.1.3.2) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:355:in `call'
config/initializers/008-rack-cors.rb:25:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/cookies.rb:689:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.1.3.2) lib/active_support/callbacks.rb:98:in `run_callbacks'
actionpack (6.1.3.2) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
logster (2.9.6) lib/logster/middleware/reporter.rb:43:in `call'
railties (6.1.3.2) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.1.3.2) lib/rails/rack/logger.rb:28:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/request_id.rb:26:in `call'
lib/middleware/enforce_hostname.rb:23:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
actionpack (6.1.3.2) lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
plugins/discourse-prometheus/lib/middleware/metrics.rb:17:in `call'
rack-mini-profiler (2.3.2) lib/mini_profiler/profiler.rb:384:in `call'


message_bus (3.3.6) lib/message_bus/rack/middleware.rb:61:in `call'
lib/middleware/request_tracker.rb:177:in `call'
rails_multisite (3.0.0) lib/rails_multisite/middleware.rb:25:in `call'
railties (6.1.3.2) lib/rails/engine.rb:539:in `call'
railties (6.1.3.2) lib/rails/railtie.rb:207:in `public_send'
railties (6.1.3.2) lib/rails/railtie.rb:207:in `method_missing'
rack (2.2.3) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.3) lib/rack/urlmap.rb:58:in `each'
rack (2.2.3) lib/rack/urlmap.rb:58:in `call'
unicorn (6.0.0) lib/unicorn/http_server.rb:634:in `process_client'
unicorn (6.0.0) lib/unicorn/http_server.rb:732:in `worker_loop'
unicorn (6.0.0) lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
unicorn (6.0.0) lib/unicorn/http_server.rb:143:in `start'
unicorn (6.0.0) bin/unicorn:128:in `<top (required)>'
vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `load'
vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `<main>'

The following line seems to be the culprit. But since the plugin is not enabled, it should not fire, right?

github.com/discourse/discourse-prometheus-alert-receiver

plugin.rb

main


      
          # frozen_string_literal: true
          
          # name: discourse-prometheus-alert-receiver
          # about: Receives a Prometheus webhook and creates a topic in Discourse
          # version: 0.1
          # url: https://github.com/discourse/discourse-prometheus-alert-receiver
          
          enabled_site_setting :prometheus_alert_receiver_enabled
          
          register_asset "stylesheets/topic-post.scss"
          
          after_initialize do
            module ::DiscoursePrometheusAlertReceiver
              PLUGIN_NAME = "discourse-prometheus-alert-receiver".freeze
          
              PREVIOUS_TOPIC_CUSTOM_FIELD = "prom_previous_topic".freeze
              TOPIC_BODY_CUSTOM_FIELD = "prom_alert_topic_body".freeze
              TOPIC_BASE_TITLE_CUSTOM_FIELD = "prom_alert_topic_base_title".freeze
          
              class Engine < ::Rails::Engine
                engine_name PLUGIN_NAME
                isolate_namespace ::DiscoursePrometheusAlertReceiver
              end
            end
          
            require_relative "app/controllers/discourse_prometheus_alert_receiver/receiver_controller"
            require_relative "app/models/alert_receiver_alert"
            require_relative "app/serializers/alert_receiver_alert"
            require_relative "app/jobs/concerns/alert_post_mixin"
            require_relative "app/jobs/regular/process_alert"
            require_relative "app/jobs/regular/process_grouped_alerts"
          
            unless Rails.env.test?
              %i[tagging_enabled allow_duplicate_topic_titles].each do |setting|
                SiteSetting.public_send("#{setting}=", true) unless SiteSetting.public_send(setting)
              end
            end
          
            register_topic_custom_field_type(
              DiscoursePrometheusAlertReceiver::PREVIOUS_TOPIC_CUSTOM_FIELD,
              :integer,
            )
            register_topic_custom_field_type(
              DiscoursePrometheusAlertReceiver::TOPIC_BODY_CUSTOM_FIELD,
              :string,
            )
            register_topic_custom_field_type(
              DiscoursePrometheusAlertReceiver::TOPIC_BASE_TITLE_CUSTOM_FIELD,
              :string,
            )
          
            self.add_model_callback("Category", :after_destroy) do
              PluginStoreRow
                .where(plugin_name: ::DiscoursePrometheusAlertReceiver::PLUGIN_NAME)
                .where("value::json->>'category_id' = '?'", self.id)
                .destroy_all
            end
          
            self.add_model_callback("Group", :after_destroy) do
              PluginStoreRow
                .where(plugin_name: ::DiscoursePrometheusAlertReceiver::PLUGIN_NAME)
                .where("value::json->>'assignee_group_id' = '?'", self.id)
                .destroy_all
            end
          
            reloadable_patch { Topic.has_many :alert_receiver_alerts, dependent: :delete_all }
          
            add_class_method(:topic, :alerts_category_ids) do
              PluginStoreRow.where(plugin_name: ::DiscoursePrometheusAlertReceiver::PLUGIN_NAME).pluck(
                "value::json->'category_id'",
              )
            end
          
            add_class_method(:topic, :firing_alerts) do |category_ids = []|
              joins(:alert_receiver_alerts).where("alert_receiver_alerts.status": "firing").where(
                "not topics.closed AND topics.category_id IN (?)",
                (category_ids.presence || alerts_category_ids),
              )
            end
          
            add_to_serializer(
              :topic_view,
              :alert_data,
              include_condition: -> { object.topic.alert_receiver_alerts.present? },
            ) { ActiveModel::ArraySerializer.new(object.topic.alert_receiver_alerts) }
          
            on(:after_extract_linked_users) do |users, post|
              if post.post_number == 1 && post.user == Discourse.system_user &&
                   post.topic.category&.slug == "alerts" # TODO: don't hardcode the category slug
                users.clear
              end
            end
          
            TopicQuery.add_custom_filter(
              DiscoursePrometheusAlertReceiver::PLUGIN_NAME,
            ) do |results, topic_query|
              options = topic_query.options
          
              if Topic.alerts_category_ids.include?(options[:category_id]) && options[:status] == "firing"
                results = results.firing_alerts([options[:category_id]])
              end
          
              results
            end
          
            require_dependency "admin_constraint"
          
            ::DiscoursePrometheusAlertReceiver::Engine.routes.draw do
              token_format = /[a-f0-9]{64}/
              post "/receiver/:token" => "receiver#receive", :token => token_format, :as => :receive
              post "/receiver/resync/:token" => "receiver#receive_grouped_alerts", :token => token_format
              post "/receiver/grouped/alerts/:token" => "receiver#receive_grouped_alerts",
                   :token => token_format,
                   :as => :receive_grouped_alerts
              post "/receiver/generate" => "receiver#generate_receiver_url",
                   :constraints => AdminConstraint.new
            end
          
            Discourse::Application.routes.append do
              mount ::DiscoursePrometheusAlertReceiver::Engine, at: "/prometheus"
            end
          end

hellekin · June 13, 2021, 9:10pm

When discourse-prometheus-alert-receiver is enabled, the site works fine. When it is not enabled, the site breaks. This sounds like a bug to me.

Topic		Replies	Views
Upgrade to 2.8.0beta11 fails on multisite due to code review plugin Product Support	0	440	January 13, 2022
Upgrade to latest release failed 8/21/25 Self-hosting	11	483	September 13, 2025
Rake multisite:migrate is broken Bug	4	1556	June 23, 2020
Exception: Failed to initialize site default after upgrade Self-hosting	3	180	April 13, 2026
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column categories.search_priority does not exist Bug	18	4744	March 25, 2019

Multi-site upgrade leaves _some_ sites unavailable

AI-generated summary

Related topics