It would be able to be unhidden using the browser inspector, though I think you could easily uncover who was doing that and moderate them accordingly.
There is an open Contribute > UX topic on removing the ability to upvote and downvote the OP. This would mean using Likes/Reactions as the positive ‘upvote’:
But if you don’t feel that covers your use case you can create a new Contribute > UX topic and suggest it there. 
Though couldn’t the downvote trolling also occur on posts as well as the OP?