# 1 million Topics - Takes millions of days to get indexed without Sitemap in Robots

**URL:** <https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234>\
**Category:** Product Support\
**Created:** [11 Febrero, 2017 00:25 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234 "2017-02-11T00:25:09Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![veer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/veer/32/69391_2.png) [@veer](https://meta.discourse.org/u/veer)\
**Post date:** [11 Febrero, 2017 00:25 UTC](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234/1 "2017-02-11T00:25:09Z")

</div>

I recently merged my multiple bbpress forums in a single Discourse Forum, Google is indexing it very very slow. It can not go very deep.

**From one month i am getting 200 pages per day get indexed, while usually a new content get indexed to 0.1 posts in a month. Its hard for google to index a big forum.**  
Following three things must be solved so that discourse can be use for multipurpose sites.

1. dofollow, noindex in header (meta- robots) of the tags pages. Forum with too many tags will create too much Duplicate content in index without this solved.

2. there must be a way to edit the robots.txt to include sitemaps, or to avoid unnecessary pages get indexed.

3. There must be a sitemap facility, just like nodebb.

In nodebb all of these three things are possible. but i used discourse as it is more stable in big databases.

---

_[View the full topic](https://meta.discourse.org/t/1-million-topics-takes-millions-of-days-to-get-indexed-without-sitemap-in-robots/57234)._