# 1 million Topics - Takes millions of days to get indexed without Sitemap in Robots
**URL:** https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234
**Category:** Support
**Created:** [11.Февраль.2017 00:25:09 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234 "2017-02-11T00:25:09Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@veer](https://meta.discourse.org/u/veer)
#### Post date: [11.Февраль.2017 00:25:09 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/1 "2017-02-11T00:25:09Z")
I recently merged my multiple bbpress forums in a single Discourse Forum, Google is indexing it very very slow. It can not go very deep.
**From one month i am getting 200 pages per day get indexed, while usually a new content get indexed to 0.1 posts in a month. Its hard for google to index a big forum.**
Following three things must be solved so that discourse can be use for multipurpose sites.
1. dofollow, noindex in header (meta- robots) of the tags pages. Forum with too many tags will create too much Duplicate content in index without this solved.
2. there must be a way to edit the robots.txt to include sitemaps, or to avoid unnecessary pages get indexed.
3. There must be a sitemap facility, just like nodebb.
In nodebb all of these three things are possible. but i used discourse as it is more stable in big databases.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [11.Февраль.2017 02:36:24 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/2 "2017-02-11T02:36:24Z")
No, none of this is true. Site maps are useless and unneeded. Turn off JavaScript and/or set your user agent to Google. See for yourself.
Every customer we have gets indexed fine, even migrations with hundreds of thousands of topics.
(The tags page issue is valid, but already fixed in 1.8 beta by @neil)
---
### Author:  [@mitchellk](https://meta.discourse.org/u/mitchellk)
#### Post date: [11.Февраль.2017 06:56:40 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/3 "2017-02-11T06:56:40Z")
Have to agree with @codinghorror every Discourse site I have seen ranks incredibly well in Google Search. Just about every technical topic I search for these days (and I search for a lot), within the top 3 results it’s almost always a site running on Discourse.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [20.Февраль.2017 16:21:29 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/4 "2017-02-20T16:21:29Z")
Recommend you try out the updates by @vinothkannans to the site map plugin and report back on the plugin thread as to how effective it was.
> [@Discourse Sitemap](https://meta.discourse.org/t/discourse-sitemap-plugin/40348):
>
> discourse2Summary Discourse Sitemap adds a sitemap.xml to your Discourse instanceopen_bookInstall Guide This plugin is bundled with Discourse core. There is no need to install the plugin separately. Features There has been [some](https://meta.discourse.org/t/how-to-create-xml-sitemap-is-there-any-plugin/33654) [discussion](https://meta.discourse.org/t/sitemap-xml-for-google-webmaster/9054/9) over the need of a sitemap, and I think Discourse doesn’t really need one, although I do believe that a forum might be indexed a bit faster with one. Anyway, we created this plugin because of a customer that needed a sitemap for Google N…
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [20.Февраль.2017 16:21:34 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/5 "2017-02-20T16:21:34Z")