How to noindex all pages to keep them out of Google indexes

Product Support

AI-generated summary

To keep pages out of Google indexes, a noindex meta tag is required on every affected page, as Google no longer supports the noindex directive in robots.txt. L30110 was looking for a way to accomplish this and pfaffman suggested adding the meta tag to all pages through a theme component by inserting <meta name="noindex"> into /admin/customize/themes/common/head_tag. However, as noted by L30110, the recommended tag to block all robots from indexing is <meta name="robots" content="noindex">.

1

I’ve seen:

but it isn’t clear what the upshot was of this discussion. A noindex directive in robots.txt is no longer effective – Google now ignores it. To keep a private site from appearing in search results entirely, Disallow is not sufficient. What’s needed is the noindex meta tag on every affected page. In our case, this is the login page and any error pages externally reachable (or, for that matter, a noindex meta tag for every page would be just fine.

How can this be accomplished? Thanks.

2

The “upshot” or outcome of that discussion appears to be very clear:

  • a new feature was added to add noindex for pages that shouldn’t be crawled
  • that new feature had to be removed/reverted because Google didn’t support it and was then reporting robots.txt as invalid.
3

Yes, Google stopped supporting noindex in robots.txt. But their recommendation now appears to be putting a noindex meta tag on all pages that you want to be entirely eliminated from their index. That’s what I’m trying to accomplish.

4

Can’t you just make a theme component that will add the noindex meta tag to all pages?

Just add

<meta name="noindex">

to /admin/ customize/ themes/ common/ head_tag

Right?

5

Sorry, I should have been more explicit that the first part of the topic post should be removed.

You are probably better to refer to the Google announcement because that has a link to the page HTML option that pfaffman suggests above:
Block search indexing with ‘noindex’

6

I haven’t done much theme work, so this is exactly what I needed to know. Thanks! And it’s showing up in the head correctly (added to “head_tag”). For the record, the Google recommended tag if you want to ask all robots not to index, is:

<meta name="robots" content="noindex">

I’ve now disabled (well, actually commented out) robots.txt completely for the reasons noted earlier in this thread. Thanks again.

7

Glad I could help! I’m still not very good at themes and know nothing about SEO, so I didn’t know what the meta tag actually needed to look like. Glad I got you close enough for you to solve it.