At the moment it is gated on trust level:
poll minimum trust level to create can be raised to a higher TL if you want less people to have access.
Medium term we would certainly like to change it so poll create permission is group based.
This feels like a duplicate of: Restrict specific group to create poll ? Would you like to make a specific feature request here?