Most likely an issue between your hosting provider and the concerned ISP. Check with your hosting provider if they have a way to identify inbound connectivity.
Getting a traceroute from affected users might be helpful in identifying where the connections are being dropped.