Discourse Sitemap Plugin

(Sam Saffron) #61

I guess the problem is that it is never able to even render first time around.

grabbing 50K topics can take a while, I would try a pluck there instead of a select cause it would be much faster

Short of that a site setting to limit the size cause maybe smaller batches are needed … not sure.

2 Likes
(Veer) #62

@vinothkannans Once created Sitemaps should be stored in cached solid file form. So that big sites do not create sitemaps again and again. It will be big improvement for big discourse sites.

(Sam Saffron) #63

Not following at all, it is stored in redis which is super fast.

(Vinoth Kannan) #64

I will try the pluck and other options to minimize the the load.

3 Likes
(Veer) #65

one sitemap have more than 20 thousand posts, batch should be decreased to 5k. As most of time on 6gb ram i was browsing sitemaps, it was giving error 502 (1 in three times). This occurs on already browsed sitemap too. (which we are thinking, that it is cached.)

(Vinoth Kannan) #66

Okay. Then I will try to catch the whole sitemap output instead of caching the topics data. And the above suggestion of @sam about site setting.

2 Likes
(Vinoth Kannan) #67

PR created

@veer you may try out the updates when it merged and report back.

  • Using pluck instead of select
  • Caching whole sitemap output instead of topics data
  • New SiteSetting to limit no. of topics per page in Sitemap
  • Default sitemap size reduced to 10000 from 50000 topics
4 Likes
(Richard - DiscourseHosting.com) #68

I have merged your PR, thank you very much.

I see one possible problem where changing the sitemap_topics_per_page setting will keep using the cached data with the old sitemap size. It might be a good idea to use the sitemap_topics_per_page value as a part of the cache key.

2 Likes
(Vinoth Kannan) #69

Good point. Created the PR

3 Likes
(Mitchell Krog) #70

Just installed this today to try it out. Looks good so far and Google reported no issues with the format. Good work guys.

3 Likes
(Mitchell Krog) #71

Ok already found an issue. Had someone comment on an article today and the sitemap shows the article as last updated on the 17th of March ???

<loc>
https://photographic.community/t/revisiting-old-images-and-re-processing-them-6-years-later/92
</loc>
<lastmod>2017-03-17T10:59:14Z</lastmod>
(Vinoth Kannan) #72

Currently system generating sitemap once in 24 hours only. So it is not a bug. You are viewing cached output. It will be changed in next 24 hours.

2 Likes
(Mitchell Krog) #73

Excellent, thanks so much for the explanation.

Xml sitemap with ssl forum
(Hosein Naseri) #74

I’m using sitemap plugin with my forum that I prefer to be loaded with https. However the http protocol is also available and not turned off. How can I tell the sitemap plugin to add the ssl version of urls? currently all links in the sitemap begin with http.

(Rafael dos Santos Silva) #75

By default a lot of Discourse internals will use HTTP until you check force_https. You should do that if your site works on HTTPS you shouldn’t allow HTTP anymore.

1 Like
(Hosein Naseri) #76

But I want the http to be available. because I have implemented the forum in my android application, However since older android versions have some issues with some ssl protocols, I do my api calls with http protocol. I don’t know if i enable force_https, am I able to still use http protocol in my api calls?

(Rafael dos Santos Silva) #77

Oh I see that’s a pretty strict requirement. Looks like in your case there is no way out.

Those are Androids below 4.3 ?

(Hosein Naseri) #78

I think so.

Since I’m really fine with not force_https ing my forum, I think an option for this plugin to list ssl urls would be the best thing for me.

(Vinoth Kannan) #79

I guess you should customize this plugin little bit for your requirements.

3 Likes
(Richard - DiscourseHosting.com) #80

Note: It seems like the caching mechanism is broken in Rails 5.

Looking into it later today, trying to get a fix out by today or tomorrow.
In the meanwhile: input and suggestions are welcome :slight_smile:
Details here https://github.com/discoursehosting/discourse-sitemap/issues/5

EDIT found it. render :text is deprecated in Rails 5 - should be render :plain instead. Pushed a fix.

10 Likes