I don’t think that’s the entire story. I believe your attachments are using filetypes that the browser wants to load natively, like .html or .js.
Your easy solution is to enable a CDN, they are quite inexpensive these days, and offer much better performance as the required JS, CSS, etc is all served from a geographical location much closer to the user, wherever they are in the world.