That is not necessarily true, an installed but disabled plugin can indeed cause performance concerns, see for instance Disabled plugins still causing performance impact
Now that specific issue has been solved (for the most?) on the bundled plugins, but on other plugins this might still be happening here and there.