From your description, my guess would be that your server is resolving ipv6 records when making DNS lookups (we use getaddrinfo for that). But then when attempting to connect to them, it’s failing.
That might be because it tries the ip6 address, fails, and then falls back to the ip4 address. We don’t currently have that kind of fallback logic in FinalDestination#resolve: