Expected behaviour of `solved` replies when plugin is switched off?

Yes, because the plugin is still active within Discourse, all disable does is remove the UI elements which allow you to enable solved against categories and mark posts as solved.

To disable it you need to comment it out in your app.yml and rebuild.