This is a networking problem between your VPS host (Bloom) and RubyGems CDN. There are several reports where this is caused by traffic between Level3 and Fastly and a failure to properly detect the packet size for this route.
A user reported success working around this issue with lowering the host MTU: