Posso instalar o discourse neste vps não gerenciado? Se sim, como?

You need this link: Install Discourse in production with the official, supported instructions

There are many providers of VPS. Consider Digital Ocean, Hetzner, Scaleway too.

Checkout discussions in hosting