Sidekiq fails to start after Zeitwerk on docker dev

averi · October 2, 2019, 10:44am

We recently upgraded to 2.4.0.beta5 from 2.4.0.beta4 and sidekiq fails to start with the following traceback:

> $ bundle exec sidekiq -C config/sidekiq.yml
> uninitialized constant SiteSetting::SiteSettingExtension
> /discourse/app/models/site_setting.rb:5:in `<class:SiteSetting>'
> /discourse/app/models/site_setting.rb:3:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/zeitwerk-2.1.10/lib/zeitwerk/kernel.rb:23:in `require'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/dependencies/interlock.rb:13:in `loading'
> /discourse/config/initializers/004-message_bus.rb:120:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:667:in `block in load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/activesupport-6.0.0/lib/active_support/notifications.rb:182:in `instrument'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:666:in `load_config_initializer'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/engine.rb:623:in `block in <class:Engine>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `instance_exec'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:32:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:61:in `block in run_initializers'
> /usr/local/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:50:in `tsort_each_child'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `call'
> /usr/local/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
> /usr/local/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
> /usr/local/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/initializable.rb:60:in `run_initializers'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/application.rb:363:in `initialize!'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `public_send'
> /discourse/vendor/bundle/ruby/2.5.0/gems/railties-6.0.0/lib/rails/railtie.rb:190:in `method_missing'
> /discourse/config/environment.rb:7:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
> /discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
> /discourse/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:74:in `kernel_load'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli/exec.rb:28:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:463:in `exec'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:27:in `dispatch'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/cli.rb:18:in `start'
> /usr/local/bin/bundle:30:in `block in <main>'
> /usr/local/lib/ruby/site_ruby/2.5.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
> /usr/local/bin/bundle:22:in `<main>'

Any guidance on where to look next? Thanks!

Falco · October 2, 2019, 12:14pm

This isn’t a install that followed our official guide, so there isn’t much we can help you with.

averi · October 2, 2019, 12:27pm

We’re currently running discourse on Openshift so surely it’s not a standard install, I’ll review the official guide and see whether I missed anything on the current deployment, thanks.

azawawi · October 2, 2019, 12:39pm

Actually even the following docker command is triggering the same stack trace in my discourse vagrant machine over virtualbox:

vagrant ssh -c '(cd ~/discourse && sudo bin/docker/sidekiq)'

Falco · October 2, 2019, 12:42pm

You sure about that? The official guide doesn’t have anything related manual starts of sidekiq and there is no /home/discourse folder in it.

Are you running a development install in production ??

azawawi · October 2, 2019, 12:45pm

I am using the docker commands to test and develop discourse plugins.

The other script I am currently using is:

# Clear temporary discourse cache and launch development rails
vagrant ssh -c '(rm -rf ~/discourse/tmp/cache)'
vagrant ssh -c '(cd ~/discourse && sudo bin/docker/rails s)'

azawawi · October 2, 2019, 12:47pm

Of course not

azawawi · October 2, 2019, 12:50pm

@Falco I think it is related to

https://github.com/discourse/discourse/commit/427d54b2b00fa94474c0522eaed750452c4e7f43#diff-b2f320bf18d0c321021eb4a8f6704169

What do you think?

Falco · October 2, 2019, 12:51pm

Oh it may be. The fact that you didn’t state that this is your development setup and not a dev environment didn’t help.

So you are saying that our docker dev setup is currently broken?

azawawi · October 2, 2019, 12:52pm

Yes this looks like it.

azawawi · October 2, 2019, 12:59pm

Reverting that commit makes sidekiq work again.

Falco · October 2, 2019, 1:11pm

Looks like this is one for you @kris.kotlarek

kris.kotlarek · October 2, 2019, 11:32pm

Ok, I know what happened here, it is my fault. I removed a lot of require_dependency since they are not required anymore when using Zeitwerk autoloader.

However, in application.rb we got that:

if !Sidekiq.server?
  config.autoload_paths += Dir["#{config.root}/lib"]
end

Which means that Sidekiq is not looking for lib directory to find dependencies and we explicitly define what is required in specific files.

I can bring back that require_depenedency for files which are used by Sidekiq or remove that guard in application.rb.

I guess that we used that explicit require to save some memory for workers, so probably we should follow that path. I will bring back require_dependency

@sam what is your opinion?

sam · October 2, 2019, 11:36pm

We should remove the guard, I don’t like it cause you have no idea if your process will become a sidekiq or not. In our deployments we have: unicorn master -> fork -> sidekiq worker. At fork time application.rb was already parsed.

azawawi · October 3, 2019, 7:18am

@kris.kotlarek Thanks for the fix. Sidekiq is now working along with the discourse official plugins .

Most of my 3rd-party plugins with jobs are not working now . Most likely caused by the following commit not being applied on their Jobs:: classes.

https://github.com/discourse/discourse-assign/commit/d59e7fe1fbe95789324010b3729d0589a8a9789f#diff-5428237aebbe478a7a57cc3d42db4bb1

Broken 3rd-party plugins

https://github.com/gdpelican/babble/blob/master/app/jobs/regular/babble_post_alert.rb#L2

https://github.com/procourse/procourse-static-pages/blob/master/app/jobs/onceoff/migrate_static_pages_plugin.rb#L2

https://github.com/discourse/discourse-whos-online/blob/master/plugin.rb#L110

kris.kotlarek · October 3, 2019, 11:07pm

You are right, all Jobs::Onceoff, Jobs::Base and Jobs::Scheduled require now ::

I fixed discourse-whos-online and created pull requests for other plugins:
https://github.com/gdpelican/babble/pull/287
https://github.com/procourse/procourse-static-pages/pull/11

azawawi · October 4, 2019, 5:13am

@kris.kotlarek Thanks a lot . I took a look at your post for the zeitwork auto-loader. Is there a plan to enable a development plugin auto-reloading feature to speed up plugin development.

kris.kotlarek · October 4, 2019, 5:33am

There are still places containing require or require_dependency but many are removed.

I think that now plugin’s code should be automatically reloaded without a problem. We need to play a little bit to ensure, but I have a positive feeling

azawawi · October 6, 2019, 9:21am

It seems we need to enable it first as per its documentation. I tried it in development and no plugin changes were reloaded

https://github.com/fxn/zeitwerk#reloading

fzngagan · October 6, 2019, 9:23am

This would be great. I had some hope but your words give me more hope. Plugin devs will love this.

Topic		Replies	Views
Sidekiq not running installation	21	4996	June 19, 2017
Upgrading Discourse to Zeitwerk dev	14	3158	October 2, 2019
Sidekiq heartbeat test failed, restarting installation	12	1790	February 11, 2020
Unable to start sidekiq, load_missing_constant dev	2	1505	August 25, 2017
Is it possible to use a local plugin directory with a docker install? dev	39	2027	June 16, 2022

Sidekiq fails to start after Zeitwerk on docker dev

Broken 3rd-party plugins

Related Topics