It would be able to be unhidden using the browser inspector, though I think you could easily uncover who was doing that and moderate them accordingly.
There is an open ux topic on removing the ability to upvote and downvote the OP. This would mean using Likes/Reactions as the positive ‘upvote’:
But if you don’t feel that covers your use case you can create a new ux topic and suggest it there. 
Though couldn’t the downvote trolling also occur on posts as well as the OP?