Hi everyone, We just realised that our Discourse forum is not indexed by Google (we remember that it was indexed about a year ago), and we’re trying to fix it right now. What are the configuration that we need to make sure are set properly? This is what I’ve done so far: I’ve made sure that “al…

Admin -> Settings -> Enable Robots.txt [image] Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt Login to Google Webmaster Tools and check: https://www.google.com/webmasters/tools/robots-testing-tool

Out of the box with all defaults this works totally fine, did you modify these settings when you originally installed?

[صورة] tohaitrieu: Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt The robots.txt file has this text in the middle, so it might have problems with crawlers: User-agent: * Disallow: / Noindex: / Google is indexing pages though: https://www.google.com/search?q=site%3Aht…

Yes. Access: https://discuss.grakn.ai/admin/customize/robots Remove: User-agent: * Disallow: / Noindex: / Go to Google Webmaster Tools: https://www.google.com/webmasters/tools/robots-testing-tool Choose a verified property and submit robots.txt again to Google. I think it should work…

Finally, removing the following block fixed the problem. User-agent: * Disallow: / Noindex: / Thank you so much, @j127 and @tohaitrieu !!! Google Search Console now shows that discuss.grakn.ai is queued up for indexing. Cheers!

I’m very unclear how you ended up in this state. Did you change default site settings related to crawling?

I’m also unclear how we ended up in the above state, @codinghorror . I’ve been the admin of the site for the past year and I did not change anything related to stuff above. I do remember not doing an upgrade for very long, and then did one shortly before the above issue started occurring, but I don’t…

ملف robots.txt غير صالح يتسبب في مشاكل بالفهرسة

الدعم

tohaitrieu (To Trieu) 19 أغسطس 2019، 1:01م 2

Admin -> Settings -> Enable Robots.txt

Your Forum Roboy file is allowed: https://discuss.grakn.ai/robots.txt

الموضوع		الردود	مرات العرض
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	236	18 ديسمبر 2024
Google changed how they process robots.txt in Discourse? Support	20	1746	22 ديسمبر 2020
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3332	30 يوليو 2019
Generic rules in "robots.txt" not picked up by Googlebot Support	6	1017	2 أبريل 2022
Google notification to remove "noindex" statements from robots.txt Support	8	2467	30 يوليو 2019

ملف robots.txt غير صالح يتسبب في مشاكل بالفهرسة

الموضوعات ذات الصلة