Hi, first of all, thank you very much for the new RSS Polling admin interface. It makes such a difference!
We have a website with dozens of RSS feeds. I have noticed in Sidekiq that some jobs for some feeds may take twenty minutes and more. This easily clogs the queue.
There is a setting RSS polling feed request timeout but this is not the problem. The RSS feed is quickly available and it loads fast (at least when I try with my browser at the same time that Sidekiq is struggling with the same feed). The problem seems to be that some RSS feeds contain all the episodes of a podcast (as opposed to just the 10-15 most recent) AND that Discourse seems to process all those entries even if the episodes are already downloaded and without changes. Basically, Sidekiq can be easily struggling for half hour or more due to RSS feeds that bring no changes, no new episodes.
Ideally, Discourse would quickly see that there are no changes and move on.
Optionally, there could be a setting to limit the reading to the first NN entries of the feed.
Or at the very least, a time limit to end a polling if it isn’t resolved after NN minutes (5 minutes, or 10 at most)
I wonder what exactly happens when a feed takes more than 10 minutes to be processed and Sidekiq is 100% busy. What exactly can take that long and be so heavy to process.
Do you think this is something that could be improved in the current plugin?
i’d be happy to improve this, would you be able to share the url of your feed so that i can better test locally? (feel free to pm me if you want it kept private)
Thank you very much @zogstrip for your quick reply! The RSS feeds are public, and in fact recommended for whoever likes to listen independent podkasts and radio programs in Spanish.
Many of them use WordPress and we can ask them to edit the number of entries in their RSS. But many use iVoox, a popular platform at least in Spanish speaking countries. They ship all the episodes of a program in their RSS, they are a (relatively) big company, and it is difficult to even find a “Contact us” link that will lead to someone. So we are stuck there. For your test, I would start adding the iVoox feeds, and then go for more if you want more.
I’m very happy of our little contribution reporting this problem. Even if our case is quite a corner case (I guess?), the solution will contribute at a smaller degree to the performance of basically any Discourse instance using this plugin.
I see the patches are merged. Does this mean that we can get them by updating Discourse now?
Merged means that the changes are in main. Usually, forums don’t track the main branch; instead, they track latest. This means that after merging, you still have to wait for the automatic tests to run and pass.
When you are looking at Commits · discourse/discourse · GitHub, you can see that this has just happened with zogstrip’s commits (there is a green check mark, while the commit above still has the brown dot).
I have reenabled the RSS feeds and the first RSS round has mostly gone through. 86 new posts have appeared, many of them from days ago, which means that several feeds were not able to complete their processing before, and now they have. Very good!
Very long feeds still can keep a job busy for a long time, but I have only seen 3 going beyond 10 minutes. This one is still running after 1 hour… But it is really long, so maybe it is justified? Then again, a single feed taking one job for so long is kind of problematic in itself. Maybe it has to do with being the first full import in a while? I will check again in the next RSS round (we have it set to 180 minutes).
Maybe that job running during one hour is stuck on something beyond the import? I have just noticed that all the posts of that feed seem to have been imported, with images and everything. At least from the user point of view, the import seems to be done. Maybe it’s an outlier. I will report again after a couple more RSS rounds.
are those feeds “new” or have they all been already imported? i haven’t done much when it comes to improving how fast we do the initial import as this is largely limited by how “fast” we can create new topic.
Ok, we have changed the polling frequency to 10 minutes for the sake of testing (we will bring it back to 180 minutes, good enough for us). After zogstrip’s fix, most feeds are processed within a minute or less. There is only a couple of very long feeds that can get to 2 minutes and beyond, but never (that I have seen) beyond 5 minutes. This RSS feed is the only one that reliably gets to 4 minutes or more, but it is extremely long, so I’m not sure much can be done about this use case. Even the browser needs while to just render it. (And we might remove it for unrelated reasons.)
The fact that the entire polling process doesn’t take more than 5-6 minutes is astounding. A huge improvement. For me, this report can be resolved. Thank you very much again, @zogstrip.
