That is a bit… over positive. For testing purposes it can be enough, and it can be a little bit tight even then. 2GB is much closer to reality.
4 GB is enough for bigger instances, though. But amount and power of cores plays important role too.
And still… if you have 5 GB and rebuilding takes that long time there must be something wrong.
I’m on DigitalOcean/4 GB/2 intel vcpu and rebuilding takes something around 25 minutes.
What is your system exacly, plus app.yml — is there something custom?