Yes, we use Docker Hub and I know that sometimes it can be a bit slow.
A workaround you can use is to pre-download the base image before issuing the rebuild, with something like:
docker pull $(grep -o "discourse/base:[0-9.-]*" launcher)
Maybe when GitHub Docker Registry becomes usable we can also host the image in there.