Sure, but which of those 4 billion IPs is the right one if you can’t tell them apart?
With proper opsec (e.g. firewall rules to only allow Cloudflare in and whitelisted traffic out) means you will be pretty safe. Until you aren’t because you missed a leak 
For example, better proxy all your DNS as well.