Sidekiq fails to start after Zeitwerk on docker dev

We recently upgraded to 2.4.0.beta5 from 2.4.0.beta4 and sidekiq fails to start with the following traceback:

> $ bundle exec sidekiq -C config/sidekiq.yml
> uninitialized constant SiteSetting::SiteSettingExtension
> /discourse/app/models/site_setting.rb:5:in `<class:SiteSetting>'
> /discourse/app/models/site_setting.rb:3:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:13:in `loading'
> /discourse/config/initializers/004-message_bus.rb:120:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:667:in `block in load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/notifications.rb:182:in `instrument'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:666:in `load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `block in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `instance_exec'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:61:in `block in run_initializers'
> /usr/local/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `tsort_each_child'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:60:in `run_initializers'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/application.rb:363:in `initialize!'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `public_send'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `method_missing'
> /discourse/config/environment.rb:7:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
> /usr/local/bin/bundle:30:in `block in <main>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
> /usr/local/bin/bundle:22:in `<main>'

Any guidance on where to look next? Thanks!

1 Like

This isn’t a install that followed our official guide, so there isn’t much we can hel you with.

1 Like

We’re currently running discourse on Openshift so surely it’s not a standard install, I’ll review the official guide and see whether I missed anything on the current deployment, thanks.

1 Like

Actually even the following docker command is triggering the same stack trace in my discourse vagrant machine over virtualbox:

vagrant ssh -c '(cd ~/discourse && sudo bin/docker/sidekiq)'

You sure about that? The official guide doesn’t have anything related manual starts of sidekiq and there is no /home/discourse folder in it.

Are you running a development install in production :exploding_head:??

1 Like

I am using the docker commands to test and develop discourse plugins.

The other script I am currently using is:

# Clear temporary discourse cache and launch development rails
vagrant ssh -c '(rm -rf ~/discourse/tmp/cache)'
vagrant ssh -c '(cd ~/discourse && sudo bin/docker/rails s)'

Of course not :smile:

@Falco I think it is related to

What do you think?

1 Like

Oh it may be. The fact that you didn’t state that this is your development setup and not a dev environment didn’t help.

So you are saying that our docker dev setup is currently broken?

1 Like

Yes this looks like it.

Reverting that commit makes sidekiq work again.

1 Like

Looks like this is one for you @kris.kotlarek

8 Likes

Ok, I know what happened here, it is my fault. I removed a lot of require_dependency since they are not required anymore when using Zeitwerk autoloader.

However, in application.rb we got that:

if !Sidekiq.server?
  config.autoload_paths += Dir["#{config.root}/lib"]
end

Which means that Sidekiq is not looking for lib directory to find dependencies and we explicitly define what is required in specific files.

I can bring back that require_depenedency for files which are used by Sidekiq or remove that guard in application.rb.

I guess that we used that explicit require to save some memory for workers, so probably we should follow that path. I will bring back require_dependency

@sam what is your opinion?

3 Likes

We should remove the guard, I don’t like it cause you have no idea if your process will become a sidekiq or not. In our deployments we have: unicorn master -> fork -> sidekiq worker. At fork time application.rb was already parsed.

4 Likes

@kris.kotlarek Thanks for the fix. Sidekiq is now working along with the discourse official plugins :+1: .

Most of my 3rd-party plugins with jobs are not working now :cry: . Most likely caused by the following commit not being applied on their Jobs:: classes. :arrow_down:


Broken 3rd-party plugins

5 Likes

You are right, all Jobs::Onceoff, Jobs::Base and Jobs::Scheduled require now ::

I fixed discourse-whos-online and created pull requests for other plugins:


5 Likes

@kris.kotlarek Thanks a lot :partying_face: :+1: . I took a look at your post for the zeitwork auto-loader. Is there a plan to enable a development plugin auto-reloading feature to speed up plugin development.

3 Likes

There are still places containing require or require_dependency but many are removed.

I think that now plugin’s code should be automatically reloaded without a problem. We need to play a little bit to ensure, but I have a positive feeling :slight_smile:

5 Likes

It seems we need to enable it first as per its documentation. I tried it in development and no plugin changes were reloaded :cry:

1 Like

This would be great. I had some hope but your words give me more hope. Plugin devs will love this.

1 Like