If you search here for “reactions” you’ll find a few interesting topics. The first is about an official plugin which I’m sure you could use for downvotes - you could have a 
reaction icon (see Discourse Reactions).
If you just want downvotes you could change the 
to (see Change the Like Icon to 👍)!