We don’t have a way only to show the summarize button to a specific subset of users. The following is a site setting to enable the creation of “new” summaries.
Certainly, something for us to keep in mind as we give control of access for other AI features.