We use the rss-polling plugin to import entries from dozens of feeds. From time to time, some feeds will just vanish from the list. We don’t know under which conditions, we don’t know how, and we cannot find any log of such action. We haven’t found a pattern either. They just vanish, and we need to create their new entries again (when we realize they have disappeared, which isn’t trivial with so many feeds).
Is there a functionality written in the code that would eliminate an RSS feed under certain conditions? Is there a log that would register when this happens?
The only thing we do with RSS feed is to create new ones manually on /admin/plugins/rss_polling. Nothing else.
I have noticed that the order of RSS feeds in that page changes from time to time, which is already strange and it doesn’t help debugging problems because the chronological order of feed creation is lost, and the new sorting doesn’t seem to follow any alphabetical or numerical sorting.
We don’t know how often this problem happens because we cannot detect it right away when it happens. In a best case scenario we can only wait for the owner of the feed to tell us that new entries are not appearing, or we realize ourselves that, say, a weekly podcast has been silent on the forum lately.
I have the impression that this problem might be related to situations where the CPU gets close to 100%, or something else in the server reaching maximum capacity. Even if the scheduler does a good job at polling only 5 feeds at a time, on the hosting analytics we can see that when RSS feeds are being updated the CPU basically reaches 100%.
Maybe this is why forums with just a couple of feeds don’t notice it, but if you add plenty something might break only sometimes regardless of the scheduler?
In any case, I wonder why Discourse should delete RSS feeds in the first place, and do it silently. If a RSS feed is problematic for some reason, it could skip it this one time and log the problem. From this point two things could happen:
It was an occasional problem and the next scheduler runs will work just fine. The log entry will be forgotten.
The problem persists, every skip leaves an entry in the error log, and when someone realizes there is a problem the admins can check the error log and find the details.
Ideally admins would get some kind of notification, but I understand if this adds more work to a potential solution. Skip instead of delete + error log entry would be a big improvement already.
And to explain what impact this problem has in our project. We have a portal of independent podcasts, they contact us to be aggregated, we do create a channel (subcategory) where their programs are imported and where listeners can like and comment… It gives a very bad impression about our project and the Discourse platform when they realize we have stopped aggregating them and we have to answer them that somehow their feed has been deleted…
Let’s start with this fix for now. I’m not saying this is the cause of feeds mysteriously being deleted, but adding at least a confirmation dialog box before actually deleting a feed is a start.
I’ll work later on more improvements when I have more time. I want to add a proper edit/save individual feeds UI instead of just updating/saving ALL of them each edit.
@icaria36 if you update your plugin you will get this latest change. I’ll follow up again when I have more improvements ready.
This “temporary” code that has been here over a year is likely the culprit. I’ll take a look.