Pages listed in the robots.txt are crawled and indexed by Google

Huh, it looks like noindex may be supported in robots.txt already?

And it seems it works

Ultimately, the NoIndex directive in Robots.txt is pretty effective. It worked in 11 out of 12 cases we tested. It might work for your site, and because of how it’s implemented it gives you a path to prevent crawling of a page AND also have it removed from the index.

cc @sam this will be the easiest way.

6 Likes