Hi everyone, We just realised that our Discourse forum is not indexed by Google (we remember that it was indexed about a year ago), and we’re trying to fix it right now. What are the configuration that we need to make sure are set properly? This is what I’ve done so far: I’ve made sure that “al…

Admin -> Settings -> Enable Robots.txt [image] Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt Login to Google Webmaster Tools and check: https://www.google.com/webmasters/tools/robots-testing-tool

Out of the box with all defaults this works totally fine, did you modify these settings when you originally installed?

[画像] tohaitrieu: Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt The robots.txt file has this text in the middle, so it might have problems with crawlers: User-agent: * Disallow: / Noindex: / Google is indexing pages though: https://www.google.com/search?q=site%3Ahttp…

Yes. Access: https://discuss.grakn.ai/admin/customize/robots Remove: User-agent: * Disallow: / Noindex: / Go to Google Webmaster Tools: https://www.google.com/webmasters/tools/robots-testing-tool Choose a verified property and submit robots.txt again to Google. I think it should work…

Finally, removing the following block fixed the problem. User-agent: * Disallow: / Noindex: / Thank you so much, @j127 and @tohaitrieu !!! Google Search Console now shows that discuss.grakn.ai is queued up for indexing. Cheers!

I’m very unclear how you ended up in this state. Did you change default site settings related to crawling?

I’m also unclear how we ended up in the above state, @codinghorror . I’ve been the admin of the site for the past year and I did not change anything related to stuff above. I do remember not doing an upgrade for very long, and then did one shortly before the above issue started occurring, but I don’t…

robots.txt の不備によりインデックス作成に問題が発生

サポート

tohaitrieu (To Trieu) 2019 年 8 月 19 日午後 1:01 2

Admin -> Settings -> Enable Robots.txt

Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt

トピック		返信	表示
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	236	2024 年 12 月 18 日
Google changed how they process robots.txt in Discourse? Support	20	1746	2020 年 12 月 22 日
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3332	2019 年 7 月 30 日
Generic rules in "robots.txt" not picked up by Googlebot Support	6	1017	2022 年 4 月 2 日
Google notification to remove "noindex" statements from robots.txt Support	8	2467	2019 年 7 月 30 日

robots.txt の不備によりインデックス作成に問題が発生

関連トピック