I run many sites and this hasn’t been my experience. Have you changed anything else in that time? I’ve noticed the longest rebuild times are for sites with many plugins to download. You could try commenting them out and see if that makes a difference. 