So I just tested this out. As Stephen stated and has been previously discussed here on Meta, “turning off” a plugin via settings will not fully disable it.
I did also test removing the plugin from the app.yml and rebuilding, as expected (and I stated earlier) all evidence of the plugin is gone.