Sidekiq fails to start after Zeitwerk on docker dev

[averi (Andrea Veri)]

We recently upgraded to 2.4.0.beta5 from 2.4.0.beta4 and sidekiq fails to start with the following traceback:

> $ bundle exec sidekiq -C config/sidekiq.yml
> uninitialized constant SiteSetting::SiteSettingExtension
> /discourse/app/models/site_setting.rb:5:in `<class:SiteSetting>'
> /discourse/app/models/site_setting.rb:3:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:13:in `loading'
> /discourse/config/initializers/004-message_bus.rb:120:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:667:in `block in load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/notifications.rb:182:in `instrument'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:666:in `load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `block in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `instance_exec'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:61:in `block in run_initializers'
> /usr/local/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `tsort_each_child'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:60:in `run_initializers'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/application.rb:363:in `initialize!'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `public_send'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `method_missing'
> /discourse/config/environment.rb:7:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
> /usr/local/bin/bundle:30:in `block in <main>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
> /usr/local/bin/bundle:22:in `<main>'

Any guidance on where to look next? Thanks!

[Falco]

This isn’t a install that followed our official guide, so there isn’t much we can help you with.

[averi (Andrea Veri)]

We’re currently running discourse on Openshift so surely it’s not a standard install, I’ll review the official guide and see whether I missed anything on the current deployment, thanks.

[azawawi (Ahmad M. Zawawi)]

@Falco

This isn’t a install that followed our official guide, so there isn’t much we can hel you with.

Actually even the following docker command is triggering the same stack trace in my discourse vagrant machine over virtualbox:

vagrant ssh -c '(cd ~/discourse && sudo bin/docker/sidekiq)'

[Falco]

You sure about that? The official guide doesn’t have anything related manual starts of sidekiq and there is no /home/discourse folder in it.

Are you running a development install in production ??

[azawawi (Ahmad M. Zawawi)]

I am using the docker commands to test and develop discourse plugins.

The other script I am currently using is:

# Clear temporary discourse cache and launch development rails
vagrant ssh -c '(rm -rf ~/discourse/tmp/cache)'
vagrant ssh -c '(cd ~/discourse && sudo bin/docker/rails s)'

[azawawi (Ahmad M. Zawawi)]

@Falco

Are you running a development install in production ??

Of course not

[azawawi (Ahmad M. Zawawi)]

@Falco I think it is related to

https://github.com/discourse/discourse/commit/427d54b2b00fa94474c0522eaed750452c4e7f43#diff-b2f320bf18d0c321021eb4a8f6704169

What do you think?

[Falco]

Oh it may be. The fact that you didn’t state that this is your development setup and not a dev environment didn’t help.

So you are saying that our docker dev setup is currently broken?

[azawawi (Ahmad M. Zawawi)]

Yes this looks like it.

[azawawi (Ahmad M. Zawawi)]

@Falco

So you are saying that our docker dev setup is currently broken?

Reverting that commit makes sidekiq work again.

[Falco]

Looks like this is one for you @kris.kotlarek

[kris.kotlarek (Kris)]

Ok, I know what happened here, it is my fault. I removed a lot of require_dependency since they are not required anymore when using Zeitwerk autoloader.

However, in application.rb we got that:

if !Sidekiq.server?
  config.autoload_paths += Dir["#{config.root}/lib"]
end

Which means that Sidekiq is not looking for lib directory to find dependencies and we explicitly define what is required in specific files.

I can bring back that require_depenedency for files which are used by Sidekiq or remove that guard in application.rb.

I guess that we used that explicit require to save some memory for workers, so probably we should follow that path. I will bring back require_dependency

@sam what is your opinion?

[sam (Sam Saffron)]

@kris.kotlarek

I can bring back that require_depenedency for files which are used by Sidekiq or remove that guard in application.rb .

We should remove the guard, I don’t like it cause you have no idea if your process will become a sidekiq or not. In our deployments we have: unicorn master → fork → sidekiq worker. At fork time application.rb was already parsed.

[azawawi (Ahmad M. Zawawi)]

@kris.kotlarek Thanks for the fix. Sidekiq is now working along with the discourse official plugins .

Most of my 3rd-party plugins with jobs are not working now . Most likely caused by the following commit not being applied on their Jobs:: classes.

https://github.com/discourse/discourse-assign/commit/d59e7fe1fbe95789324010b3729d0589a8a9789f#diff-5428237aebbe478a7a57cc3d42db4bb1

---

Broken 3rd-party plugins

https://github.com/gdpelican/babble/blob/master/app/jobs/regular/babble_post_alert.rb#L2

https://github.com/procourse/procourse-static-pages/blob/master/app/jobs/onceoff/migrate_static_pages_plugin.rb#L2

https://github.com/discourse/discourse-whos-online/blob/master/plugin.rb#L110

[kris.kotlarek (Kris)]

You are right, all Jobs::Onceoff, Jobs::Base and Jobs::Scheduled require now ::

I fixed discourse-whos-online and created pull requests for other plugins:
https://github.com/gdpelican/babble/pull/287
https://github.com/procourse/procourse-static-pages/pull/11

[azawawi (Ahmad M. Zawawi)]

@kris.kotlarek Thanks a lot . I took a look at your post for the zeitwork auto-loader. Is there a plan to enable a development plugin auto-reloading feature to speed up plugin development.

Upgrading Discourse to Zeitwerk

Rails 6 ships with two autoloading modes: zeitwerk and classic. In that pull request https://github.com/discourse/discourse/pull/8083 I upgraded Rails to version 6.0.0 with classic autoloader as a transitional phase. It would be interesting to try to switch to Zeitwerk. Zeitwerk is an efficient and thread-safe code loader for Ruby. As long as the project is following naming conventions, Zeitwerk can find correct files and load them on demand or upfront without the need for any require or requir…

GitHub - fxn/zeitwerk: Efficient and thread-safe code loader for Ruby

[kris.kotlarek (Kris)]

There are still places containing require or require_dependency but many are removed.

I think that now plugin’s code should be automatically reloaded without a problem. We need to play a little bit to ensure, but I have a positive feeling

[azawawi (Ahmad M. Zawawi)]

It seems we need to enable it first as per its documentation. I tried it in development and no plugin changes were reloaded

https://github.com/fxn/zeitwerk#reloading

Autoloading and Reloading Constants (Zeitwerk Mode) — Ruby on Rails Guides

[fzngagan (Faizaan Gagan)]

This would be great. I had some hope but your words give me more hope. Plugin devs will love this.