# תוסף Sitemap - אין כתובות ?page=… במפת האתר של ברירת המחדל

**URL:** https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283
**Category:** Bug
**Tags:** sitemap
**Created:** [7 במרץ,‏ 2022,‏ 1:35pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283 "2022-03-07T13:35:11Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![rrit](https://avatars.discourse-cdn.com/v4/letter/r/b5ac83/32.png) [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [7 במרץ,‏ 2022,‏ 1:35pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/1 "2022-03-07T13:35:11Z")

</div>

The #sitemap plugin does not include any `?page=…` urls in **default** sitemaps e.g. [https://meta.discourse.org/sitemap\_4.xml](https://meta.discourse.org/sitemap_4.xml)

```xml
<url>
  <loc>
    https://meta.discourse.org/t/importing-migrating-from-phpbb3/30810
  </loc>
  <lastmod>2022-02-25T21:55:40Z</lastmod>
</url>

```

In the **recent** sitemap the pagination urls are included - e.g. [https://meta.discourse.org/sitemap\_recent.xml](https://meta.discourse.org/sitemap_recent.xml)

```xml
<url>
  <loc>
    https://meta.discourse.org/t/importing-migrating-from-phpbb3/30810?page=18
  </loc>
  <lastmod>2022-03-07T12:03:50Z</lastmod>
</url>

```

Are there no `?page=…` urls in **default** sitemaps by design?  
All these `?page=…` urls are canonical urls and thereby should be added to the default sitemap - e.g.

```xml
<url>
  <loc>
    https://meta.discourse.org/t/importing-migrating-from-phpbb3/30810
  </loc>
  <lastmod>2022-02-25T21:55:40Z</lastmod>
</url>
<url>
  <loc>
    https://meta.discourse.org/t/importing-migrating-from-phpbb3/30810?page=2
  </loc>
  <lastmod>2022-03-02T19:08:07Z</lastmod>
</url>

[…]

<url>
  <loc>
    https://meta.discourse.org/t/importing-migrating-from-phpbb3/30810?page=18
  </loc>
  <lastmod>2022-03-07T12:03:50Z</lastmod>
</url>

```

* * *

### sources

> <https://github.com/discourse/discourse-sitemap/blob/099d074690c2118e158a8baa63c3ea16b56819f1/plugin.rb#L45-L56>

> <https://github.com/discourse/discourse-sitemap/blob/099d074690c2118e158a8baa63c3ea16b56819f1/plugin.rb#L58-L68>

> <https://github.com/discourse/discourse-sitemap/blob/099d074690c2118e158a8baa63c3ea16b56819f1/app/views/discourse_sitemap/sitemap/default.erb#L13-L18>

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [8 במרץ,‏ 2022,‏ 12:54am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/2 "2022-03-08T00:54:01Z")

</div>

I do not think this is deliberate, can you do a PR to fix?

---

<div class="post-metadata">

### Author: ![mstm](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [11 במרץ,‏ 2022,‏ 8:37pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/3 "2022-03-11T20:37:45Z")

</div>

Thanks @rrit, a few months ago I noticed this too but I always thought it was normal 🤦🏻‍♂️

---

<div class="post-metadata">

### Author: ![rrit](https://avatars.discourse-cdn.com/v4/letter/r/b5ac83/32.png) [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [13 במרץ,‏ 2022,‏ 7:29pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/4 "2022-03-13T19:29:25Z")

</div>

I can implement an easy fix which is not very specific about the last edited date: e.g all pages of one topic will use the same date of the last edited post.  
Thereby on a new post in a topic (with many posts and many pages) all the pages will get a new last changed date - even when only the last page needs the new date.

Is this a feasable solution?

* * *

Otherwise we need to bundle all posts of a topic in packages of 20 posts (per page). And then calculate the last changed date for each package itself.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [13 במרץ,‏ 2022,‏ 11:54pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/5 "2022-03-13T23:54:21Z")

</div>

Honestly I looked at this and I am mixed on any changes here, the issue is not that Google is having trouble discovering content on Discourse forums.

It is that it is discovering, crawling and then due to “arbitrary decision making” deciding that content does not belong in the index.

---

<div class="post-metadata">

### Author: ![rrit](https://avatars.discourse-cdn.com/v4/letter/r/b5ac83/32.png) [@rrit](https://meta.discourse.org/u/rrit)
#### Post date: [1 באפריל,‏ 2022,‏ 9:23am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/6 "2022-04-01T09:23:32Z")

</div>

> [@sam](#):
>
> It is that it is discovering, crawling and then due to “arbitrary decision making” deciding that content does not belong in the index.

Does “arbitrary decision making” imply one of these points:  
(See [Page indexing report - Search Console Help](https://support.google.com/webmasters/answer/7440203#crawled))

> [@](#):
>
> - **[Crawled](https://support.google.com/webmasters/answer/7643010) - currently not indexed:** The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.
> 
> - **Discovered - currently not indexed:** The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.

* * *

On the pro side of adding all these `?page=…` canonical urls to the sitemap:  
It gives Google a strong hint on `<lastmod>` for these url. Thereby Google has no reason to re-crawl unchanged `?page=…` urls and might use its precious crawl budget for more important urls.

If `?page=…` urls are missing in the sitemap, Google finds them anyway and does some “arbitrary” re-crawling - even if it’s totally unnecessary as there are no new changes to the content.

See [Build and Submit a Sitemap | Google Search Central &nbsp;|&nbsp; Documentation &nbsp;|&nbsp; Google for Developers](https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap?hl=en#general-guidelines)

> [@](#):
>
> - **Google uses the `<lastmod>` value** if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.

* * *

Google _really_ keeps track and makes a difference of where it knows about urls from:  
“All submitted pages” (sitemap) or “All known pages” (links etc.)  
See [Google Search Console → Index → Coverage Report](https://search.google.com/search-console/index?hl=en)

- “A sitemap is an important way for Google to discover URLs on your site.” [see](https://developers.google.com/search/docs/advanced/crawling/ask-google-to-recrawl#sitemap)
- “Google chooses the canonical page based on a number of factors (or _signals_ ), such as […], presence of the URL in a sitemap, […].” [see](https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls?hl=en#how-googlebot-indexes-and-chooses-the-canonical-url)
- “Using a sitemap doesn’t guarantee that all the items in your sitemap will be crawled and indexed, as Google processes rely on complex algorithms to schedule crawling.” [see](https://developers.google.com/search/docs/advanced/sitemaps/overview?hl=en#do-i-need-a-sitemap)

---

<div class="post-metadata">

### Author: ![mstm](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@mstm](https://meta.discourse.org/u/mstm)
#### Post date: [3 באפריל,‏ 2022,‏ 10:26am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/7 "2022-04-03T10:26:43Z")

</div>

> [@rrit](#):
>
> The #sitemap plugin does not include any `?page=…` urls in **default** sitemaps e.g.

I hope it will be implemented along with this 🙂

[https://github.com/discourse/discourse-sitemap/pull/34](https://github.com/discourse/discourse-sitemap/pull/34)

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [4 באפריל,‏ 2022,‏ 5:24am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/8 "2022-04-04T05:24:11Z")

</div>

This is certainly something for @Roman to keep in mind when he integrates this into core.

I much prefer first merging in sitemap prior to layering more changes, but once that is done … maybe we can start with canonical page based urls on \_recent. We have canonical url now which is usable in `posts.rss` with adequate caching it can also be usable in sitemaps.

---

<div class="post-metadata">

### Author: ![rahim123](https://avatars.discourse-cdn.com/v4/letter/r/df705f/32.png) [@rahim123](https://meta.discourse.org/u/rahim123)
#### Post date: [18 באוקטובר,‏ 2023,‏ 6:38am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/10 "2023-10-18T06:38:44Z")

</div>

I’m having trouble with Google Search Console trying to index URLs like `https://example.com/t/title-slug/1234?page=3` , which make Discourse throw a 404. Removing the `?page=x` parameter makes the URL valid.

I assume this is some kind of a side effect of Discourse adding pagination to the version of the site that it serves crawlers:

> [@?page= bug, both in core and in sitemap plugin](https://meta.discourse.org/t/page-bug-both-in-core-and-in-sitemap-plugin/191719/1):
>
> As soon as a topic has 18 posts, the `sitemap_recent.xml` starts showing a URL with a page number `https://forum.example.com/t/slug/123?page=2`

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [18 באוקטובר,‏ 2023,‏ 7:06am UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/11 "2023-10-18T07:06:11Z")

</div>

page urls work fine, you just need more than N posts.

Do you happen to have a ton of deleted posts on said topic?

---

<div class="post-metadata">

### Author: ![rahim123](https://avatars.discourse-cdn.com/v4/letter/r/df705f/32.png) [@rahim123](https://meta.discourse.org/u/rahim123)
#### Post date: [18 באוקטובר,‏ 2023,‏ 1:47pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/12 "2023-10-18T13:47:55Z")

</div>

> [@sam](#):
>
> Do you happen to have a ton of deleted posts on said topic?

Hi Sam, thanks for the reply. After posting this I found your explanation here:

> [@Googlebot 404 errors due to page numbers](https://meta.discourse.org/t/googlebot-404-errors-due-to-page-numbers/96337/9):
>
> That topic was split so you destabilized the page number, it is a fundamental limitation with our numbering implementation Take a book analogy, in a book if you rip out page 77, page 200 remains page 200. In Discourse page 200 becomes page 199 I guess, maybe it does make sense to auto redirect high page numbers back to last page.

But in my case, no, the topics with this problem that I’ve looked at don’t show any modifications to the original threading. The only thing is that they were imported from Drupal. But I need to dig more into other examples to see if any topics that were originally created in Discourse are also affected, because unfortunately there are _tons_ of them, as in thousands probably.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [18 באוקטובר,‏ 2023,‏ 10:41pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/13 "2023-10-18T22:41:56Z")

</div>

Yikes, were tons imported from Drupal? Is that the common thread here?

---

<div class="post-metadata">

### Author: ![rahim123](https://avatars.discourse-cdn.com/v4/letter/r/df705f/32.png) [@rahim123](https://meta.discourse.org/u/rahim123)
#### Post date: [18 באוקטובר,‏ 2023,‏ 10:44pm UTC](https://meta.discourse.org/t/sitemap-plugin-no-page-urls-in-default-sitemap/220283/14 "2023-10-18T22:44:18Z")

</div>

Yeah, close to 100k topics and ~2M posts. I’m not sure if this issue is only with imported topics though, I’ll post back here soon if I find any more anomalies.
