How to disable direct https://ipaddress visit?

i just want when people visit the website via https://serverip he will get an internal 500 page or nothing at all and cant go on by clicking trust the certification. so i am seaching for the solution on nginx or firewall. any sugguestion?