We are building a site relying heavily on RSS Polling (a community of podcasts, each with their own category and their own RSS publishing new posts as they publish new episodes in their platforms). I was wondering what is the risk of us dying of success, as in bringing more feeds than what the RSS Polling can handle. Specifically: Have you tested this plugin with a very high number of feeds? If there are many feeds, does it make a difference in terms of stress to poll every 10 minutes or, say, every hour? Are all the feeds being polled at the same time, are they polled in sequence, or something else? I’m asking because a simultaneous poll would be in theory more sensitive to a high amount of feeds than if feeds are being polled one after the other. A recurrent and so far harmless error maybe related to having several feeds has motivated me to ask these questions now rather than when it’s too late. :slight_smile:

Almost as expected, this question is becoming more relevant as we add more feeds to our site. Right now, when we add a new feed, it is almost guaranteed that the first import won’t happen within the 10 minutes expected (according to the polling frequency defined in the settings). Also, the sorting of feeds in the admin page where all the feeds are defined… it can change. With fewer feeds it was always a static list sorted by the same order as the feeds were added. I don’t know what causes the change of order, nor what principle feeds follow in the newer sortings. I just wonder whether these are just symptoms of a bigger problem that will melt down our site or make the polling/posting of feeds unreliable. [image] icaria36: Are all the feeds being polled at the same time, are they polled in sequence, or something else? I think this is the most relevant question. If someone can point me to the piece of code instructing the polling, maybe I can help myself. [image] icaria36: A recurrent and so far harmless error maybe related Just an update, we learned that this problem was unrelated to the amount of feeds. And it is solved now. Very good!

Also interested in the performance related questions here

Sidekiq offers some answers. Today I’ve noticed that the pages took a bit too much to load. In the last days I already noticed some sluggishness here and there, for instance in the time that it took to Discourse links to render as page titles. I checked Sidekiq and there were +100 jobs Enqueued + 5 Busy. Almost all of them RSS Polling feeds. I changed the polling periodicity from 10 minutes to 1 hour and deleted the jobs. Then I checked the server, and this is what happened before and after the change: [image] My guess (based on this alone, I still haven’t checked the code) is that RSS Polling will send the feeds to the queue, as many as you have. The queue will ensure that there is a reasonable number of simultaneous jobs. But… I guess the risk is that the queue of feeds becomes so long that when the new RSS Poll is engaged, the last one is still running, and this is when things get ugly until there is a reset, but the queue will grow again, and etc. Please correct me if this analysis is wrong. :slight_smile: UPDATE: RSS Polling in action, +60 feeds (on a still new community without masive activity and even less during sleeping hours). [image]

Hi, did you manage to know if there is a maximum number of feeds you can use ?Thanks.

Hi, I’m very very sorry for this very very late reply. I had missed the question. Just in case this is still interesting to someone… There is no hard limit as far as I can tell, but the limit is on how powerful your server is, how long the RSS feeds (as in number of episodes, probably how heavy the images?), and how often you pull. RSS polling still sends sidekiq to the roof, some feeds may take 20 minutes and more to process, and if you 5 simultaneous jobs as clog by these long RSS pulls then the job queue starts to grow. The real limit is when Sidekiq doesn’t have enough time to process all jobs in all queues between RSS polls. When that happens then the next RSS poll is likely to clog the queues further and eventually something will crash/restart. The user might not notice anything. Some RSS feeds might not be updated because of this.

Are there any upper limits to the RSS Polling plugin?

Product Support

icaria36 December 20, 2024, 9:23am 5

Related? Maybe: RSS feeds disappearing misteriously

Topic		Replies	Views
RSS Polling may take 20 minutes and more to process a feed UX rss-polling , fixed	10	399	September 16, 2026
Rss polling frequency Product Support rss-polling	3	197	June 12, 2025
Is it possible to limit polling frequency per feed? Product Support rss-polling	1	198	February 3, 2021
A new admin experience for RSS Polling Announcements rss-polling	5	390	July 24, 2026
RSS feeds no longer updating Product Support rss , rss-polling	4	246	March 29, 2025

Are there any upper limits to the RSS Polling plugin?

Related topics