Manual rebuilds which require downtime happen once or twice a year. If you upgrade via /admin/upgrade the upgrades are seamless.
You can significantly reduce your rebuild times with a two-container install and I would recommend you look into this regardless of whether you use nginx.