Closing the loop here. We decided not to add any more attributes to the directory serializer. Instead, we made vast improvements to the performance of user serialization. We split user cards onto their own route, and added a new route which returns multiple cards in one go.
You can find an example of using this new route in this theme component: