How to noindex all pages to keep them out of Google indexes

[L30110]

I’ve seen:

Pages listed in the robots.txt are crawled and indexed by Google

Pages meant to be hidden from Google are in the robots.txt However, Google attempts to crawl them anyway. Since they are accessible through links on web pages, they get indexed. The crawler then hits the robots.txt and crawl “blank”/“blocked”. This results in a lot of error in our analytics console and wasted crawls. The simple solution would be to use a noindex meta tag. Could there be a way to add this on a page by page basis? [image]

but it isn’t clear what the upshot was of this discussion. A noindex directive in robots.txt is no longer effective – Google now ignores it. To keep a private site from appearing in search results entirely, Disallow is not sufficient. What’s needed is the noindex meta tag on every affected page. In our case, this is the login page and any error pages externally reachable (or, for that matter, a noindex meta tag for every page would be just fine.

How can this be accomplished? Thanks.

[Remah (Just another happy Discourse user)]

The “upshot” or outcome of that discussion appears to be very clear:

• a new feature was added to add noindex for pages that shouldn’t be crawled
• that new feature had to be removed/reverted because Google didn’t support it and was then reporting robots.txt as invalid.

[L30110]

Yes, Google stopped supporting noindex in robots.txt. But their recommendation now appears to be putting a noindex meta tag on all pages that you want to be entirely eliminated from their index. That’s what I’m trying to accomplish.

[pfaffman (Jay Pfaffman)]

Can’t you just make a theme component that will add the noindex meta tag to all pages?

Just add

<meta name="noindex">

to /admin/ customize/ themes/ common/ head_tag

Right?

[Remah (Just another happy Discourse user)]

Sorry, I should have been more explicit that the first part of the topic post should be removed.

You are probably better to refer to the Google announcement because that has a link to the page HTML option that pfaffman suggests above:
Block search indexing with ‘noindex’

[L30110]

I haven’t done much theme work, so this is exactly what I needed to know. Thanks! And it’s showing up in the head correctly (added to “head_tag”). For the record, the Google recommended tag if you want to ask all robots not to index, is:

<meta name="robots" content="noindex">

I’ve now disabled (well, actually commented out) robots.txt completely for the reasons noted earlier in this thread. Thanks again.

[pfaffman (Jay Pfaffman)]

@L30110

I haven’t done much theme work, so this is exactly what I needed to know.

Glad I could help! I’m still not very good at themes and know nothing about SEO, so I didn’t know what the meta tag actually needed to look like. Glad I got you close enough for you to solve it.