Sidekiq fails to start after Zeitwerk on docker dev

averi · October 2, 2019, 10:44am

We recently upgraded to 2.4.0.beta5 from 2.4.0.beta4 and sidekiq fails to start with the following traceback:

> $ bundle exec sidekiq -C config/sidekiq.yml
> uninitialized constant SiteSetting::SiteSettingExtension
> /discourse/app/models/site_setting.rb:5:in `<class:SiteSetting>'
> /discourse/app/models/site_setting.rb:3:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:13:in `loading'
> /discourse/config/initializers/004-message_bus.rb:120:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:667:in `block in load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/notifications.rb:182:in `instrument'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:666:in `load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `block in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `instance_exec'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:61:in `block in run_initializers'
> /usr/local/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `tsort_each_child'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:60:in `run_initializers'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/application.rb:363:in `initialize!'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `public_send'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `method_missing'
> /discourse/config/environment.rb:7:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
> /usr/local/bin/bundle:30:in `block in <main>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
> /usr/local/bin/bundle:22:in `<main>'

Any guidance on where to look next? Thanks!

Falco · October 2, 2019, 12:14pm

This isn’t a install that followed our official guide, so there isn’t much we can help you with.

averi · October 2, 2019, 12:27pm

We’re currently running discourse on Openshift so surely it’s not a standard install, I’ll review the official guide and see whether I missed anything on the current deployment, thanks.

azawawi · October 2, 2019, 12:39pm

Actually even the following docker command is triggering the same stack trace in my discourse vagrant machine over virtualbox:

vagrant ssh -c '(cd ~/discourse && sudo bin/docker/sidekiq)'

Falco · October 2, 2019, 12:42pm

You sure about that? The official guide doesn’t have anything related manual starts of sidekiq and there is no /home/discourse folder in it.

Are you running a development install in production ??

azawawi · October 2, 2019, 12:45pm

I am using the docker commands to test and develop discourse plugins.

The other script I am currently using is:

# Clear temporary discourse cache and launch development rails
vagrant ssh -c '(rm -rf ~/discourse/tmp/cache)'
vagrant ssh -c '(cd ~/discourse && sudo bin/docker/rails s)'

azawawi · October 2, 2019, 12:47pm

Of course not

azawawi · October 2, 2019, 12:50pm

@Falco I think it is related to

https://github.com/discourse/discourse/commit/427d54b2b00fa94474c0522eaed750452c4e7f43#diff-b2f320bf18d0c321021eb4a8f6704169

What do you think?

Falco · October 2, 2019, 12:51pm

Oh it may be. The fact that you didn’t state that this is your development setup and not a dev environment didn’t help.

So you are saying that our docker dev setup is currently broken?

azawawi · October 2, 2019, 12:52pm

Yes this looks like it.

azawawi · October 2, 2019, 12:59pm

Reverting that commit makes sidekiq work again.

Falco · October 2, 2019, 1:11pm

Looks like this is one for you @kris.kotlarek

kris.kotlarek · October 2, 2019, 11:32pm

Ok, I know what happened here, it is my fault. I removed a lot of require_dependency since they are not required anymore when using Zeitwerk autoloader.

However, in application.rb we got that:

if !Sidekiq.server?
  config.autoload_paths += Dir["#{config.root}/lib"]
end

Which means that Sidekiq is not looking for lib directory to find dependencies and we explicitly define what is required in specific files.

I can bring back that require_depenedency for files which are used by Sidekiq or remove that guard in application.rb.

I guess that we used that explicit require to save some memory for workers, so probably we should follow that path. I will bring back require_dependency

@sam what is your opinion?

sam · October 2, 2019, 11:36pm

We should remove the guard, I don’t like it cause you have no idea if your process will become a sidekiq or not. In our deployments we have: unicorn master → fork → sidekiq worker. At fork time application.rb was already parsed.

azawawi · October 3, 2019, 7:18am

@kris.kotlarek Thanks for the fix. Sidekiq is now working along with the discourse official plugins .

Most of my 3rd-party plugins with jobs are not working now . Most likely caused by the following commit not being applied on their Jobs:: classes.

Broken 3rd-party plugins

github.com/gdpelican/babble

app/jobs/regular/babble_post_alert.rb

master


      
          module Jobs
            class BabblePostAlert < ::Jobs::Base
              def execute(args)
                return unless post = Post.find_by(id: args[:post_id])
                return unless post.topic&.archetype == Archetype.chat
                Babble::PostAlerter.new.after_save_post(post)
              end
            end
          end

github.com/procourse/procourse-static-pages

app/jobs/onceoff/migrate_static_pages_plugin.rb

master


      
          module Jobs
            class MigrateStaticPagesPlugin < ::Jobs::Onceoff
              # Migrate content from dl_static_pages name to procourse_static_pages
              def execute_onceoff(args)
                  dl_page_presence = PluginStoreRow.where(plugin_name: "dl_static_pages").exists?
                  if dl_page_presence
                      # Migrate the page content rows
                      PluginStoreRow.where(plugin_name: 'dl_static_pages')
                      .find_each do |row|
                          PluginStore.set("procourse_static_pages", row.key, row.value)
                          row.destroy # clean up dl_static_pages row
                      end

github.com/discourse/discourse-whos-online

plugin.rb

main


      
          # frozen_string_literal: true
          
          # name: discourse-whos-online
          # about: Display a list of online users at the top of the screen
          # meta_topic_id: 52345
          # version: 2.0
          # authors: David Taylor
          # url: https://github.com/discourse/discourse-whos-online
          
          enabled_site_setting :whos_online_enabled
          
          register_asset "stylesheets/whos_online.scss"
          
          after_initialize do
            module ::DiscourseWhosOnline
              CHANNEL_NAME = "/whos-online/online"
            end
          
            register_presence_channel_prefix("whos-online") do |channel_name|
              if channel_name == "/whos-online/online"
                config = PresenceChannel::Config.new(timeout: SiteSetting.whos_online_active_timeago * 60)
                if SiteSetting.whos_online_display_public
                  config.public = true
                else
                  config.allowed_group_ids = [::Group::AUTO_GROUPS[:trust_level_0]]
                end
                config.count_only = SiteSetting.whos_online_count_only
                config
              end
            end
          
            on(:user_seen) do |user|
              hidden = false
              hidden ||= user.user_option.hide_presence if defined?(user.user_option.hide_presence)
              hidden ||= user.id < 0
              next if hidden
              PresenceChannel.new(DiscourseWhosOnline::CHANNEL_NAME).present(
                user_id: user.id,
                client_id: "seen",
              )
            rescue PresenceChannel::InvalidAccess
            end
          
            add_to_serializer(
              :site,
              :whos_online_state,
              include_condition: -> do
                @whos_online_channel ||= PresenceChannel.new(DiscourseWhosOnline::CHANNEL_NAME)
                @whos_online_channel.can_view?(user_id: scope.user&.id)
              end,
            ) do
              @whos_online_channel ||= PresenceChannel.new(DiscourseWhosOnline::CHANNEL_NAME)
              PresenceChannelStateSerializer.new(@whos_online_channel.state, root: nil)
            end
          end

kris.kotlarek · October 3, 2019, 11:07pm

You are right, all Jobs::Onceoff, Jobs::Base and Jobs::Scheduled require now ::

I fixed discourse-whos-online and created pull requests for other plugins:

azawawi · October 4, 2019, 5:13am

@kris.kotlarek Thanks a lot . I took a look at your post for the zeitwork auto-loader. Is there a plan to enable a development plugin auto-reloading feature to speed up plugin development.

kris.kotlarek · October 4, 2019, 5:33am

There are still places containing require or require_dependency but many are removed.

I think that now plugin’s code should be automatically reloaded without a problem. We need to play a little bit to ensure, but I have a positive feeling

azawawi · October 6, 2019, 9:21am

It seems we need to enable it first as per its documentation. I tried it in development and no plugin changes were reloaded

fzngagan · October 6, 2019, 9:23am

This would be great. I had some hope but your words give me more hope. Plugin devs will love this.

Topic		Replies	Views
Upgrading Discourse to Zeitwerk Dev dev-news	14	3346	October 2, 2019
Sidekiq not working Installation	10	1114	October 2, 2023
"Ensure sidekiq is running." when it is definitely running Installation	19	7762	October 24, 2015
Development environment : running sidekiq and discourse at startup Dev	14	5527	October 30, 2018
Sidekiq not running. Sidekiq heartbeat test failed, restarting Installation unsupported-install	15	2795	June 10, 2020

Broken 3rd-party plugins

Related topics