Sitemap: `lastmod` for sitemaps wrong

Contribute Bug
3

It’s not about moving topics into different sitemap-chunks. The topics can stay in the same sitemap-chunk where they are already in.
(The mapping topic-to-sitemap-chunk is arbitrary anyway as the db select-statement with limit has no order defined.)

The bug report is about that the lastmod date of each sitemap-chunk should represent the lastmod date of the latest topic which the sitemap-chunk contains.

The way for Google should be:

  1. Load sitemap.xml
    → Check lastmod of sitemap-chunks and queue sitemap-chunks which need an update
    (lastmod date is newer than last time downloaded)

  2. Load queued sitemap-chunks sitemap_[1-5].xml
    → Check lastmod of topic-urls and queue topic-urls which need an update
    (lastmod date is newer than last time downloaded)

  3. Load queued topic-urls.

If in sitemap.xml the lastmod of the sitemap-chunks is wrong:
→ Google does not queue changed sitemaps-chunks (step 1)
→ Google does not update changed sitemap-chunks in a timely manner (step 2)
→ Google does not update changed topics in a timely manner (step 3)

Right now https://meta.discourse.org/sitemap.xml looks like this:

  • https://meta.discourse.org/sitemap_1.xml
    lastmod: 2024-03-19T12:50:09Z
    All topics inside have older or same date? :github_check:

    • Latest topic: https://meta.discourse.org/t/creating-a-stickypost-for-forum-threads/299967
      lastmod: 2024-03-19T11:03:38Z

  • https://meta.discourse.org/sitemap_2.xml
    lastmod: 2024-03-16T12:59:17Z
    All topics inside have older or same date? :x:

    • Latest topic: https://meta.discourse.org/t/launcher-rebuild-app-error-bootstrap-failed-with-exit-code-125/299538
      lastmod: 2024-03-19T09:17:46Z

  • https://meta.discourse.org/sitemap_3.xml
    lastmod: 2024-03-16T12:59:17Z
    All topics inside have older or same date? :x:

    • Latest topic: https://meta.discourse.org/t/configure-direct-delivery-incoming-email-for-self-hosted-sites/49487
      lastmod: 2024-03-18T18:16:26Z

  • https://meta.discourse.org/sitemap_4.xml
    lastmod: 2024-03-16T12:59:17Z
    All topics inside have older or same date? :x:

    • Latest topic: https://meta.discourse.org/t/video-thumbnails-issue/263595
      lastmod: 2024-03-19T00:00:20Z

  • https://meta.discourse.org/sitemap_5.xml
    lastmod: 2024-03-16T12:59:17Z
    All topics inside have older or same date? :x:

    • Latest topic: https://meta.discourse.org/t/daily-summary-9pm-utc/291850
      lastmod: 2024-03-18T21:14:49Z

  • https://meta.discourse.org/sitemap_recent.xml
    2024-03-19T13:03:41Z
    All topics inside have older or same date? :github_check:

    • Latest topic: https://meta.discourse.org/t/daily-summary-1pm-utc/291852
      lastmod: 2024-03-19T13:02:07Z