Automatically refreshing the page is handled by ember CLI. At the moment, theme and plugin assets are compiled and served by the Rails app, not Ember CLI. Therefore theme/plugin changes don’t automatically refresh the page.
This might be something we can improve in the future. But for now, it’s expected behaviour.