We are building a site relying heavily on RSS Polling (a community of podcasts, each with their own category and their own RSS publishing new posts as they publish new episodes in their platforms). I was wondering what is the risk of us dying of success, as in bringing more feeds than what the RSS Polling can handle.
Specifically:
- Have you tested this plugin with a very high number of feeds?
- If there are many feeds, does it make a difference in terms of stress to poll every 10 minutes or, say, every hour?
- Are all the feeds being polled at the same time, are they polled in sequence, or something else? I’m asking because a simultaneous poll would be in theory more sensitive to a high amount of feeds than if feeds are being polled one after the other.
A recurrent and so far harmless error maybe related to having several feeds has motivated me to ask these questions now rather than when it’s too late. 
3 Likes
Almost as expected, this question is becoming more relevant as we add more feeds to our site.
Right now, when we add a new feed, it is almost guaranteed that the first import won’t happen within the 10 minutes expected (according to the polling frequency defined in the settings).
Also, the sorting of feeds in the admin page where all the feeds are defined… it can change. With fewer feeds it was always a static list sorted by the same order as the feeds were added. I don’t know what causes the change of order, nor what principle feeds follow in the newer sortings.
I just wonder whether these are just symptoms of a bigger problem that will melt down our site or make the polling/posting of feeds unreliable.
I think this is the most relevant question. If someone can point me to the piece of code instructing the polling, maybe I can help myself.
Just an update, we learned that this problem was unrelated to the amount of feeds. And it is solved now. Very good!
1 Like
Also interested in the performance related questions here
2 Likes
Sidekiq offers some answers.
Today I’ve noticed that the pages took a bit too much to load. In the last days I already noticed some sluggishness here and there, for instance in the time that it took to Discourse links to render as page titles.
I checked Sidekiq and there were +100 jobs Enqueued + 5 Busy. Almost all of them RSS Polling feeds. I changed the polling periodicity from 10 minutes to 1 hour and deleted the jobs.
Then I checked the server, and this is what happened before and after the change:
My guess (based on this alone, I still haven’t checked the code) is that RSS Polling will send the feeds to the queue, as many as you have. The queue will ensure that there is a reasonable number of simultaneous jobs. But… I guess the risk is that the queue of feeds becomes so long that when the new RSS Poll is engaged, the last one is still running, and this is when things get ugly until there is a reset, but the queue will grow again, and etc.
Please correct me if this analysis is wrong.
UPDATE: RSS Polling in action, +60 feeds (on a still new community without masive activity and even less during sleeping hours).
