For sites that don’t want this, I strongly recommend simply uploading a: “Robots not welcome” text file
The content is 100% customisable and will continue to be.
404ing this, cause of reasons is not really making sense to me, but if someone really need to do that a plugin is possible.