The datacenter, he.net does offer remote hands for $100/hour. So if it is very urgent I would call them, they would disconnect and reconnect all the network cables to the hot spare secondary switch in the same order. Pretty easy, since both our live and hot spare are the exact same switch and stacked right on top of each other.
If I had to drive down, it is about an hour to get there. (Berkeley to San Jose)
The main things that fail are hard drives and power supplies. Failure for new, burned-in server hardware is not that common… I never saw any failures at all for the ~10 servers we built up in the 3 years after deployed server hardware for Stack Exchange.
However, in my experience, while you are getting the servers initially set up and configured, you will need physical access a LOT in the beginning. Not because things are failing, but because you always forget something in the configuration. After racking the servers, plan for a few weeks of visiting the datacenter once a week. Once that is over, you’ll barely ever go back.
(and IPMI aka remote KVM-over-IP works amazingly well, you can reboot and edit the BIOS over the internet… as long as the server has power, it can be managed using IPMI which is basically a dedicated little ARM computer with its own networking inside the server.)
Just my experience!