How to noindex all pages to keep them out of Google indexes

I’ve seen:

but it isn’t clear what the upshot was of this discussion. A noindex directive in robots.txt is no longer effective – Google now ignores it. To keep a private site from appearing in search results entirely, Disallow is not sufficient. What’s needed is the noindex meta tag on every affected page. In our case, this is the login page and any error pages externally reachable (or, for that matter, a noindex meta tag for every page would be just fine.

How can this be accomplished? Thanks.

3 Likes

The “upshot” or outcome of that discussion appears to be very clear:

  • a new feature was added to add noindex for pages that shouldn’t be crawled
  • that new feature had to be removed/reverted because Google didn’t support it and was then reporting robots.txt as invalid.
2 Likes

Yes, Google stopped supporting noindex in robots.txt. But their recommendation now appears to be putting a noindex meta tag on all pages that you want to be entirely eliminated from their index. That’s what I’m trying to accomplish.

3 Likes

Can’t you just make a theme component that will add the noindex meta tag to all pages?

Just add

<meta name="noindex">

to /admin/ customize/ themes/ common/ head_tag

Right?

3 Likes

Sorry, I should have been more explicit that the first part of the topic post should be removed.

You are probably better to refer to the Google announcement because that has a link to the page HTML option that pfaffman suggests above:
Block search indexing with ‘noindex’

3 Likes

I haven’t done much theme work, so this is exactly what I needed to know. Thanks! And it’s showing up in the head correctly (added to “head_tag”). For the record, the Google recommended tag if you want to ask all robots not to index, is:

<meta name="robots" content="noindex">

I’ve now disabled (well, actually commented out) robots.txt completely for the reasons noted earlier in this thread. Thanks again.

5 Likes

Glad I could help! I’m still not very good at themes and know nothing about SEO, so I didn’t know what the meta tag actually needed to look like. Glad I got you close enough for you to solve it.

6 Likes