Disabled plugins still causing performance impact

[loic (Loïc Guitaut)]

I just finished a PR making most of the `register_*` calls behave like add_to_serializer:

https://github.com/discourse/discourse/pull/35630