Just an update to this, I figured it would be interesting to see if I could find a workaround without moving server.
I’ve successfully monkeypatched Oneboxer and Onebox::Helpers in a local plugin to call a proprietary third party API to bring back the target page source reliably if the direct response is nil (ie my request was rejected). There is a (very) small charge to my API account every time I call it so it needs to be used sparingly for those sites that are tricky.
The advantage is I can remain on a cheap-as-chips server whilst not having to worry about its ‘reputation’ (a bit like using a 3rd party mail service).
The code still works as normal for sites that don’t reject my first request so I’m not charged for the majority of previews.
I anticipate that the costs of using the third party API will be much lower than the increase in costs for moving to a VPS provider with a more trusted reputation.
But in anycase problem solved. Thanks to everyone who weighed in.
This is now a working plugin.