Check the mail header to see where the delays are originating. It will show you which hops are reached at what times.