Hi everyone, We just realised that our Discourse forum is not indexed by Google (we remember that it was indexed about a year ago), and we’re trying to fix it right now. What are the configuration that we need to make sure are set properly? This is what I’ve done so far: I’ve made sure that “allow index in robots txt” is ticked I’ve added the following domains to “exclude rel nofollow domains”: grakn.ai (our main site domain) discuss.grakn.ai (our discourse forum domain) I’ve made sure that “add rel nofollow to user content” is unticked I’ve added Googlebot to “whitelisted crawler user agents” Am I missing any other configurations that I need to set? Our Google Search Console shows that discuss.grakn.ai could still not be crawled because it is blocked by robots.txt - see screenshot below. Thanks in advance for the help!!! [28]

Admin → Settings → Enable Robots.txt [image] Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt Login to Google Webmaster Tools and check: https://www.google.com/webmasters/tools/robots-testing-tool

[image] tohaitrieu: Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt The robots.txt file has this text in the middle, so it might have problems with crawlers: User-agent: * Disallow: / Noindex: / Google is indexing pages though: Google Search It might be that Googlebot is looking at your Google-specific rules and Webmaster Tools is warning you about the wildcard. (I’m not sure what settings result in that robots.txt output.)

Yes. Access: https://discuss.grakn.ai/admin/customize/robots Remove: User-agent: * Disallow: / Noindex: / Go to Google Webmaster Tools: https://www.google.com/webmasters/tools/robots-testing-tool Choose a verified property and submit robots.txt again to Google. I think it should work.

Finally, removing the following block fixed the problem. User-agent: * Disallow: / Noindex: / Thank you so much, @j127 and @tohaitrieu!!! Google Search Console now shows that discuss.grakn.ai is queued up for indexing. Cheers!

I’m very unclear how you ended up in this state. Did you change default site settings related to crawling?

I’m also unclear how we ended up in the above state, @codinghorror. I’ve been the admin of the site for the past year and I did not change anything related to stuff above. I do remember not doing an upgrade for very long, and then did one shortly before the above issue started occurring, but I don’t know if that’s related.

Malformed robots.txt causing issues with indexing

Product Support

codinghorror (Jeff Atwood) August 20, 2019, 3:21am 3

Out of the box with all defaults this works totally fine, did you modify these settings when you originally installed?

Topic		Replies	Views
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Product Support	18	435	December 18, 2024
Google changed how they process robots.txt in Discourse? Product Support	20	2020	December 22, 2020
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3616	July 30, 2019
Generic rules in "robots.txt" not picked up by Googlebot Product Support	5	1127	March 3, 2022
Google notification to remove "noindex" statements from robots.txt Product Support	7	2636	July 30, 2019

Malformed robots.txt causing issues with indexing

Related topics