The ‘supported’ way is to disable the plugins via the admin UI.
You’re free to delete them on the filesystem, and it’ll “probably” work (as you say, ideally plugins should be modular). But it’s not something we rigourously test at the moment, so I can’t provide a 100% guarantee on it.