TL;DR: Are there any plug-ins or settings which add a selection of random topics to the bottom of a topic (not just the most recent) even when not logged-in so that a crawler can discover all the older topics. Having now merged an older forum into the current forum, I have the curious problem of ‘surfaceability’/discoverability. In category and latest views, you can see only the most recent topics. Like the surface of a lake. The view contains only a limited number of posts and infinite scrolling makes it difficult to ‘go deep’. Having multiple categories widens the ‘straw’, but even if you have say, 50 categories and look 100 deep, that is still only 5000 topics deep. Only 1%-2% of the topics are accessible, the rest are hidden below the surface. I thought sitemaps.xml could help, but these expose only a handful of topics in the default configuration. Huge chunks of content are effectively blocked from crawlers. I was wondering, are there any plug-ins or settings which add a selection of random topics to the bottom of a topic (not just the most recent) even when not logged-in so that a crawler can discover all the older topics.

Have you considered bumping older topics? Seems like an easier way to resurface old topics without creating plugins or something. In category settings you can find these settings [image] This works well here on meta as it brings back old topics which can then be ‘updated’ with latest info / closed. Its a good way of curating your content too. Here is an example from 2020 which has been bumped [image] Anyway to only allow new user from social login? Support From the questions from Armin LI, Continuing the discussion from How to prevent registration spam: In my case, I also have the registration spam problem. That is why I want to turn off public email sign up but let users only use Github to login.

Matter of taste how good system bumping is. Or if it is done because of bots, not for humans. Here bumped topics irritates me big time. I don’t know why I should see overdated topics because of there is some not-actual need to close those.

[image] Isambard: I thought sitemaps.xml could help, but these expose only a handful of topics in the default configuration. Wait.. what? Why do you think that?

I looked at the generated sitemap and it had only half a month of posts on there in sitemap_1.xml. sitemap_recent.xml has even fewer.

And did you check sitemap_2.xml etcetera? [image] Isambard: sitemap_recent.xml has even fewer. Hence the word “recent” in the name.

[image] Richard - Communiteq: And did you check sitemap_2.xml etcetera? That’s the issue, there are no further sitemap pages other than sitemap_1.xml and that one has fewer than the 10k max URLs specified in the settings. [image] Richard - Communiteq: Hence the word “recent” in the name. I know, I mention it in case people think I’m referring to that file.

Maybe I’ll see if I can re-trigger a sitemap generation somehow.

[image] Isambard: That’s the issue, there are no further sitemap pages other than sitemap_1.xml and that one has fewer than the 10k max URLs specified in the settings. And are all those topics that are missing public? Can you post or PM the forum this is about?

The vast majority of the topics are public. I see that sitemap_1 has the max 10k urls in it (generated just under an hour ago). No other sitemap pages. I’ll wait 15 minutes when the next scheduled generation should take place to see if the others appear. I see in Sidekiq that the regeneration job did run, but I don’t know why it is producing just sitemap_1.xml and not other pages. Jobs::RegenerateSitemaps 48 minutes ago OK 257ms in 11 minutes default I can PM you the forum link if the regeneration job scheduled in 11 mins doesn’t fix it. I also bumped it to 50k urls so even if it doesn’t generate further pages, that single page will at least have 5 times the number of URLs. EDIT: Just an update. The job did run and created the new larger 50k sitemap, but again just one page. EDIT2: After letting it settle a bit, I’m happy to report the additional sitemap pages have been generated in the most recent job. I’m not sure why they didn’t generate in the earlier Jobs.

Discovering older topics: drinking a lake through a straw

Product Support

Jagster (Jakke Flemming) June 1, 2024, 9:53am 2

Crawlers should use sitemap. Not scraping randomly.

Topic		Replies	Views
Suggested topics vs. Featured topics Feature suggested-topics	10	2361	January 12, 2017
Automatically bumping old topics on a category Announcements new-feature , auto-bump	71	12698	June 18, 2021
Understanding suggested topics logic Using Discourse explanation , suggested-topics	2	1601	April 25, 2024
Easy ways to navigate and browse large categories? Product Support	4	517	June 2, 2024
Why bump old topics automatically here on Meta? Site feedback	18	4357	October 7, 2019

Discovering older topics: drinking a lake through a straw

Related topics