I moved this to support, thank you for your delightfully written bug report here.
It has taken us so much time to finely tune our rules to keep google happy x-robots-tag noindex is unevenly supported but it is an industry standard. The issue with just banning crawling was that for some reason, under certain conditions, pages could find themselves in the google index and then there was no easy way to remove them cause crawling was banned, a bit of a chicken meets egg problem.