Why tags are "disallow" in robots.txt?

It is possible to override robots.txt in this regard now, if you really want to: /admin/customize/robots.