Yes. A slow server will have slower rebuild times like this.
Some steps are, but not all. That said, a single core means sharing the CPU between the multiple processes the container have, and will be even worse when trying to serve traffic. So if you can afford it’s better to add some cores.