So making going into /admin/customize/robots Using sitemap plugin, I noticed this at the bottom. If I wanted to allow access for google to all posts that require login to view how would this part be changed? Or would I just use Allow statements for all the categories? I’m strictly looking at SEO as I changed the visibility of some categories recently and would like google to crawl the pages still. Sitemap: /sitemap.xml

It is not possible. Non-public categories require a login to be access. Google’s web-crawler does not have a login (it’s anonymous). Therefore, Google cannot see the category, or any of its content. An aside, if Google was able to access the content without a login, so could technically-knowledgeable users.

If either a category or the whole forum aren’t public, there is no way to allow Google to crawl it.

The forum is public. Some categories are not. What I am trying to accomplish is people being forced to login to view the categories but allow google to see the pages and crawl it. I was reading online with robots file that could be accomplished with Allow: but the example only showed how to allow a specific page and not a whole category.

Gotcha. Is there anyway to do the a forced popup to show the category to login on a per category basis? In sense of hiring someone to do a plugin for this?

I’m not aware of an existing plugin that allows that, but there’s definitely been discussion about this before here on Meta. A quick search should bring it up.

[image] Garrick: What I am trying to accomplish is people being forced to login to view the categories but allow google to see the pages and crawl it. Google will ban you for this behavior. Do some research.

Thanks. I saw it on an SEO blog but isn’t worth the risk. I’m just going to hide it with CSS off the main categories page.

Allow google access to logged in categories

Product Support

codinghorror (Jeff Atwood) October 26, 2019, 12:30am 7

Google will ban you for this behavior. Do some research.

Topic		Replies	Views
Crawler blocked requires login on restricted pages Product Support	2	446	June 3, 2024
Enabling google/other to crawl a 'login required' forum Product Support	3	1012	August 8, 2019
Allow crawlers to scrape and index forum but only allow access via login? Product Support	1	452	May 3, 2021
Balancing Google Indexing with Group-Based Search Restrictions Product Support	5	153	March 13, 2025
Visitors can View but not Enter? Product Support	3	576	March 8, 2020

Allow google access to logged in categories

Related topics