# Malformed robots.txt causing issues with indexing
**URL:** https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160
**Category:** Support
**Created:** [August 19, 2019, 11:13am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160 "2019-08-19T11:13:34Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@haikalpribadi](https://meta.discourse.org/u/haikalpribadi)
#### Post date: [August 19, 2019, 11:13am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/1 "2019-08-19T11:13:34Z")
Hi everyone,
We just realised that our Discourse forum is not indexed by Google (we remember that it was indexed about a year ago), and we’re trying to fix it right now. What are the configuration that we need to make sure are set properly?
This is what I’ve done so far:
1. I’ve made sure that _“allow index in robots txt”_ is **`ticked`**
2. I’ve added the following domains to _“exclude rel nofollow domains”_:
3. I’ve made sure that _“add rel nofollow to user content”_ is **`unticked`**
4. I’ve added `Googlebot` to _“whitelisted crawler user agents”_
Am I missing any other configurations that I need to set?
Our Google Search Console shows that discuss.grakn.ai could still not be crawled because it is blocked by `robots.txt` - see screenshot below.
Thanks in advance for the help!!!
---
### Author:  [@tohaitrieu](https://meta.discourse.org/u/tohaitrieu)
#### Post date: [August 19, 2019, 1:01pm UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/2 "2019-08-19T13:01:43Z")
Admin → Settings → Enable Robots.txt
Your Forum Roboy file is allowed: [https://discuss.grakn.ai/robots.txt](https://discuss.grakn.ai/robots.txt)
Login to Google Webmaster Tools and check: [https://www.google.com/webmasters/tools/robots-testing-tool](https://www.google.com/webmasters/tools/robots-testing-tool)
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [August 20, 2019, 3:21am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/3 "2019-08-20T03:21:48Z")
Out of the box with all defaults this works totally fine, did you modify these settings when you originally installed?
---
### Author:  [@j127](https://meta.discourse.org/u/j127)
#### Post date: [August 20, 2019, 5:49am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/4 "2019-08-20T05:49:18Z")
> [@tohaitrieu](#):
>
> Your Forum Roboy file is allowed: [https://discuss.grakn.ai/robots.txt](https://discuss.grakn.ai/robots.txt)
The robots.txt file has this text in the middle, so it might have problems with crawlers:
```plaintext
User-agent: *
Disallow: /
Noindex: /
```
Google is indexing pages though:
> **[Google Search](https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdiscuss.grakn.ai%2F&num=100)**
It might be that Googlebot is looking at your Google-specific rules and Webmaster Tools is warning you about the wildcard.
(I’m not sure what settings result in that robots.txt output.)
---
### Author:  [@tohaitrieu](https://meta.discourse.org/u/tohaitrieu)
#### Post date: [August 21, 2019, 4:30am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/5 "2019-08-21T04:30:11Z")
Yes.
1. Access: [https://discuss.grakn.ai/admin/customize/robots](https://discuss.grakn.ai/admin/customize/robots)
2. Remove:
3. Go to Google Webmaster Tools: [https://www.google.com/webmasters/tools/robots-testing-tool](https://www.google.com/webmasters/tools/robots-testing-tool)
Choose a verified property and submit robots.txt again to Google.
I think it should work.
---
### Author:  [@haikalpribadi](https://meta.discourse.org/u/haikalpribadi)
#### Post date: [August 21, 2019, 11:53pm UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/6 "2019-08-21T23:53:53Z")
Finally, removing the following block fixed the problem.
```plaintext
User-agent: *
Disallow: /
Noindex: /
```
Thank you so much, @j127 and @tohaitrieu!!!
Google Search Console now shows that discuss.grakn.ai is queued up for indexing.
Cheers!
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [August 22, 2019, 12:20am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/7 "2019-08-22T00:20:44Z")
I’m very unclear how you ended up in this state. Did you change default site settings related to crawling?
---
### Author:  [@haikalpribadi](https://meta.discourse.org/u/haikalpribadi)
#### Post date: [September 14, 2019, 4:46pm UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/8 "2019-09-14T16:46:44Z")
I’m also unclear how we ended up in the above state, @codinghorror. I’ve been the admin of the site for the past year and I did not change anything related to stuff above. I do remember not doing an upgrade for very long, and then did one shortly before the above issue started occurring, but I don’t know if that’s related.
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [April 21, 2023, 10:30am UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/9 "2023-04-21T10:30:27Z")
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [April 21, 2023, 12:21pm UTC](https://meta.discourse.org/t/malformed-robots-txt-causing-issues-with-indexing/126160/10 "2023-04-21T12:21:32Z")