Structuring a plugin for Rails autoloading

Many plugins include lots of class definitions inside plugin.rb, or use require_relative to load ruby files. That works, but it comes with some disadvantages:

  1. No auto-reloading of changes in development. Any changes require a full server restart

  2. Getting the require calls in the right order can be painful

  3. If they are require’d outside the after_initialize block, then other autoloaded classes/modules may not be available

There is a solution! Plugins can lean on the standard Rails autoloading system. For new plugins, everything you need is defined in the plugin-skeleton. This topic describes how to adapt an existing plugin and extend the configuration.

1. Define a module and a Rails::Engine for your plugin

In plugin.rb, define a module for your plugin with a unique PLUGIN_NAME, and add a require_relative line to load the engine file we’re about to create.

# name: my-plugin-name
# ...

module ::MyPluginModule
  PLUGIN_NAME = "my-plugin-name"
end

require_relative "lib/my_plugin_module/engine"

Now create {plugin}/lib/my_plugin_module/engine.rb:

module ::MyPluginModule
  class Engine < ::Rails::Engine
    engine_name PLUGIN_NAME
    isolate_namespace MyPluginModule
  end
end

Important things to note:

  1. In plugin.rb, you must include :: before your module name to define it in the root namespace (otherwise, it would be defined under Plugin::Instance)

  2. require_relative "lib/.../engine" must be in the root of the plugin.rb file, not inside an after_initialize block

  3. Putting the engine in its own file under lib/ is important. Defining it directly in the plugin.rb file will not work. (Rails uses the presence of a lib/ directory to determine the root of the engine)

  4. The file path should include the module name, according to the Zeitwerk rules

  5. The engine_name is used as the prefix for rake tasks and any routes defined by the engine (:link: rails docs)

  6. isolate_namespace helps to prevent things leaking between core and the plugin (:link: Rails docs)

2. Define ruby files in the correct directory structure

The engine will now autoload all files in {plugin}/app/{type}/*. For example, we can define a controller

{plugin}/app/controllers/my_plugin_module/examples_controller.rb

module ::MyPluginModule
  class ExamplesController < ::ApplicationController
    requires_plugin PLUGIN_NAME

    def index
      render json: { hello: "world" }
    end
  end
end

This will now be autoloaded whenever anything in Rails tries to access ::MyPluginModule::MyController. To test things, try accessing that class from the rails console.

For autoloading to work correctly, file paths must match the full module/class heirarchy according to the rules defined by Zeitwerk.

3. Defining routes on the plugin’s engine

Create a {plugin}/config/routes.rb file

MyPluginModule::Engine.routes.draw do
  get "/examples" => "examples#index"
  # define routes here
end

Discourse::Application.routes.draw do
  mount ::MyPluginModule::Engine, at: "my-plugin"
end

This file will be automatically loaded by the engine, and changes will take effect without a server restart. In this case, the controller action would be available at /my-plugin/examples.json.

4. Adding more autoloaded paths

Sometimes you may like to introduce additional directories of autoloadable Ruby files. The most common example is the lib/ directory in a plugin.

Modify your engine definition to append lib/ to the engine’s autoload paths:

class Engine < ::Rails::Engine
  engine_name PLUGIN_NAME
  isolate_namespace MyPluginModule
  config.autoload_paths << File.join(config.root, "lib")
end

Now you can define a lib module like

{plugin}/lib/my_plugin_module/some_lib_module.rb

module ::MyPluginModule::SomeLibModule  
end

And now any references to ::MyPluginModule::SomeLibModule will automatically load the module from this file.

5. Profit!

All these files will now be automatically loaded without any deliberate require calls. Changes will be automatically picked up by rails and reloaded in-place with no server restart.

16 Likes

David, what prompted this documentation? An upstream change?

I can’t recall ever having to worry about maintaining a particular structure to achieve auto reload, or maybe I was just luckily following the appropriate structure? …

1 Like

No upstream change - just documenting a structure we’ve used in a few plugins (e.g. anonymous-moderators, client-performance), and would like to start using in more of our plugins.

If you accept the three disadvantages at the top of the OP, then yeah any structure will work. If you follow this structure, those problems will be resolved (and the developer experience should be much smoother!)

2 Likes

Yes, auto-reloading is very desirable.

Thanks or the clarification and the documentation!

1 Like

Also to be clear - when I say ‘autoloading’, I mean Rails’s ability (via Zeitwerk) to pull code changes into a running process almost instantly, without needing a restart. If you have load or require calls in your plugin.rb file, your plugin is almost certainly not using autoloading.

Separately, Discourse has a helper which detects changes to non-reloadable ruby files, and performs a full server restart automatically. This takes a few seconds. When this system is triggered, you’ll see something like this in the console

[DEV]: Edited files which are not autoloaded. Restarting server...
       - plugins/discourse-follow/plugin.rb
2 Likes

Ah!! That’s the distinction I was missing, thanks! I was happy just with the auto-restart, but this is really cool!

2 Likes

David,

I had an interesting issue, which I believe I’ve resolved, I wonder if you had time to comment?

I had a constant that was only being defined once in my plugin Module within /lib/locations

On every rebuild I would get the following error warning:

warning: already initialized constant Locations::REQUEST_PARTS
warning: already previous definition of REQUEST_PARTS was here

Moving this to plugin.rb like so, apparently solves the issue:

But … why?

My take is that it evaluates plugin.rb only once, but anything in /lib/plugin-module might be evaluated more than once …? But then why doesn’t it complain about other code?

1 Like

Yup exactly.

When Rails/Zeitwerk reloads code, it does remove_const :Blah, and then loads the file with the relevant name. So for your lib/locations/geocode.rb file, the autoloader is doing something like

Locations.remove_const :Geocode
load "lib/locations/geocode.rb"

So that’s why you don’t get an error about the Geocode constant being previously defined - Rails/Zeitwerk automatically remove it before reloading the file.

However, there is no way for the autoloader to know that your file was also defining Locations::REQUEST_PARTS, so it did not remove_const before loading.

So, if you wanted to keep the constant in the geocode.rb file, you could move REQUEST_PARTS inside the Geocode class (Locations::Geocode::REQUEST_PARTS)

But if you want to keep it on Locations::REQUEST_PARTS then I think moving to plugin.rb makes sense. Or, if you wanted to make it fully autoloadable, you might be able to move the module ::Locations definition, including the REQUEST_PARTS constant, to its own file like lib/locations.rb.

1 Like

That makes perfect sense, thanks!

I think there might be something else too?

Is it also fair to say that it is unnecessary to explicitly load these files in plugin.rb as in:

after_initialize do
  %w(
    ../app/models/location_country_default_site_setting.rb
    ../app/models/location_geocoding_language_site_setting.rb
    ../app/models/locations/user_location.rb
    ../app/models/locations/topic_location.rb
**    ../lib/locations/user_location_process.rb**
**    ../lib/locations/topic_location_process.rb**
**    ../lib/locations/country.rb**
**    ../lib/locations/geocode.rb**
**    ../lib/locations/helper.rb**
**    ../lib/locations/map.rb**
    ../lib/users_map.rb
    ../app/serializers/locations/geo_location_serializer.rb
    ../app/controllers/locations/geocode_controller.rb
    ../app/controllers/locations/users_map_controller.rb
  ).each do |path|
    load File.expand_path(path, __FILE__)
  end

because they are within the structure also handled by autoloading?

Yeah since /lib is included in your autoloading config, then there should be no need to manually load them :100:

2 Likes

Who doesn’t love a good code simplification?! :chefs_kiss:

1 Like