Configure the Discourse RSS Polling Plugin

[simon]

The Discourse RSS Polling plugin allows topics to be automatically created from one or more RSS feeds. This topic describes how to install and configure the plugin.

Installation

If your site is hosted by us, the plugin is pre-installed on your site, but the setting to enable it is hidden. Contact our support team to have us enable the plugin for you.

For self-hosted sites, install the plugin following the method outlined in Install Plugins in Discourse.

Configuration

For self-hosted sites, select the rss polling enabled site setting to enable the plugin. For sites on our hosting, contact out support team to have us enable the plugin for you.

The rss polling frequency site setting controls the frequency that the RSS feed is polled. The default value is to poll the feed once every 30 minutes. The setting has a minimum allowed value of 10 minutes.

Once the plugin is enabled, go to your Admin / Plugins page. Click the RSS Polling entry from the plugin’s page sidebar to add your feed URLs. Select a user from your Discourse site for the feed’s Author field. That user will become the owner of all posts published from the feed. Use the Discourse Category setting to configure which category the feed’s posts will be published to:

Note that if you have previously configured the RSS Polling plugin by setting the categories to publish to on your site’s Embedding page, those category settings will continue to work. The new method of configuring the feed categories directly on the RSS Polling page is a far more straightforward approach. Let us know if you have questions about this change.

Changing RSS Rolling Settings

Each unique RSS Feed URL will only be imported into Discourse once, and Discourse will import the entire RSS Feed at the time the RSS Polling settings for that feed are configured. Deleting and Re-adding the same Feed URL will have no impact on this functionality. Also note that an RSS Feed topic that has been deleted will not be re-imported into Discourse after it has been deleted.

You can change the author of all topics associated with a specific RSS Feed URL by changing the “Author” of the corresponding Feed URL on the RSS Polling settings page. This will cause Discourse to automatically update all topics from that RSS feed with the new author, and this will show as a system post edit event when viewed in the “Admin → Logs” page:

Changing the category of an RSS Feed URL will not automatically change the category of the corresponding topics. To change the category of topics associated with an RSS Feed you will need to manually move the topics to a new category using Administrative Bulk Operations . You will need to change the category of the RSS Feed URL on the RSS Polling settings page as well, so that all new topics being imported will go to the new category.

Changing the tags of an RSS Feed URL will also not automatically change the tag of the corresponding topics. RSS Topic tags will need to be manually changed, using Administrative Bulk Operations, similar to categories, as the tags are only applied to topics when they are first imported into Discourse.

Prevent embedded topics from being unlisted

By default, all embedded topics that are created on Discourse sites are unlisted until they receive a reply. This behavior makes sense when topics are pulled into Discourse with the javascript embed code, but it doesn’t make sense for topics that are pulled in via the RSS plugin. To prevent RSS topics from being unlisted, disable the embed unlisted site setting.

Configure the optional Feed Category Filter

The optional Feed Category Filter setting for each entry allow you to limit which feed items get published to Discourse based on the item’s category. When a category is entered into this setting, only feed items matching this category will be published to Discourse. Taking the feed at https://blog.discourse.org/feed.xml as an example, if you load that feed directly in your browser you will see that each entry has a category filter associated with it. The value of that category’s term property is what you would enter into the Feed Category Filter setting if you wished to only publish feed items for a specific category. For example:

Showing the full post content

If the embed truncate site setting is enabled on your site (it is enabled by default), feed topics will only display an excerpt of the feed post. Users can view the full post by clicking the “Show Full Post” button that is displayed below the excerpt. Discourse will then scrape the topic’s embed URL to get the full post content. If the full post content is not being correctly retrieved, you can fix the issue by configuring the embed whitelist selector site setting. See Configure the Allowed Embed Selectors Setting for details about that setting.

[a_lee (Allen Lee)]

Is there a way to set the Discourse topic author to the post author? I seem to recall some way to pull the username from the RSS but I don’t see that setting anymore (discourse embed feed key or something like that?). We use SSO login and we’d like to set the topic author properly so that replies will notify the original author.

We’re trying this plugin out in an attempt to solve the embedding errors that we began encountering sometime after February 2020… more detail at Embedding error.

[simon]

@a_lee

I seem to recall some way to pull the username from the RSS but I don’t see that setting anymore

I seem to remember that it used to be possible to pull the topic author from the feed. That functionality doesn’t exist in the RSS Polling plugin though. I’m not sure how much work it would be to have the plugin attempt to pull the creator from the feed and match them with an existing Discourse user.

[a_lee (Allen Lee)]

It looks like the embed_username_key_from_feed SiteSetting was removed here: https://github.com/discourse/discourse/commit/5f318a5241c5b641b69ef05e3494ebb281876603

I could try submitting a pull request to this plugin that adds it back as a plugin setting but might need a little hand holding

Looks like most of the work would be in feed_item.rb and poll_feed.rb as well as adding the setting to feed_setting.rb…

[simon]

Make sure you have enabled the rss polling enabled site setting. After doing that, try doing a hard refresh on your browser window. You should then see an “RSS Polling” entry in the sidebar on your Admin / Plugins page.

[simon]

I am not certain where you are having the trouble with this. When you add the feeds, your RSS Polling setting should look like this, but with a different username:

Make sure to click the Save icon. The icon used for the Save button is easy to miss. Clicking it doesn’t give the user any feedback that the settings have been saved. Possibly this is something that could be improved.

[simon]

@breetai

It’s the setting in embedding where I have the issue.

You’ll need to create two embeddable Host entries for the feed and then set the Path Allowlist to get the feed to publish to the correct category. It looks like the feed from cnn_tech.rss is served from this path: /~r/rss/cnn_tech/.* and the path for the feed at money_latest.rss is /~r/rss/money_latest/.*

Try configuring your embeddable hosts in a way similar to this to see if that gets the feed into the correct categories:

[Jonathan5]

Would it be possible to change this plugin to add Discourse tags to each RSS item in addition to category?

Thank you.

[Jonathan5]

Is it possible to use any other variables in embed.imported_from apart from %{link}?

I’d like to create a link using title and URL separately (in the default message the URL is displayed in full as a URL).

Thanks.

[Jonathan5]

When I delete topics created via RSS they always seem to rise from the dead fairly quickly. What should I do to prevent this? Thanks.

I’m going to reduce Customise/Embedding “Maximum number of posts to embed” still further in the hope that this helps.

Edit: I reduced that to 3 and deleted eight posts. All eight were simultaneously resurrected.

Maybe the answer is just not to delete a topic until it ceases to be available in the RSS feed.

==============================

31 Oct 2020: Can’t reply as I’m talking to myself here so editing the post

@Jonathan5

I’m going to reduce Customise/Embedding “Maximum number of posts to embed” still further in the hope that this helps.

I now don’t think this is relevant to the RSS plugin.

@Jonathan5

Is it possible to use any other variables in embed.imported_from apart from %{link}?

I’d still like an answer to this

@Jonathan5

Maybe the answer is just not to delete a topic until it ceases to be available in the RSS feed.

The answer was to unlist the topics so that they wouldn’t be hoovered up from the RSS feed again.

[simon]

The path that is used is set by the value of the link attribute that is set for each feed item. I think I found the paths for the feed at rss.cnn.com by adding http://rss.cnn.com/ to my site’s Feed URL setting and then looking at the embed URLs of the topics that were created by it.

[Jonathan5]

@Jonathan5

Is it possible to use any other variables in embed.imported_from apart from %{link}?

I’d like to create a link using title and URL separately (in the default message the URL is displayed in full as a URL).

I’m still interested in customising this message. Thanks in advance for any assistance.

[renato (Renato Atilio)]

I’d like to use RSS Polling to auto-post videos from YouTube playlists, which are in the form: https://www.youtube.com/feeds/videos.xml?playlist_id=ID.

However, the link attributes are the videos themselves, like https://www.youtube.com/watch?v=VID. This means I can’t use these URLs to set the categories by regex, but I could with the source/RSS URLs (each RSS would target a category). This isn’t possible right now, right?

[simon]

@renato

This isn’t possible right now, right?

It’s not currently possible. There are some details about the issue here: Discourse-rss-polling: polling YouTube playlist feed.

[renato (Renato Atilio)]

@simon

It’s not currently possible.

Ok, so just to be clear: it isn’t possible to use YouTube feeds because they use some custom media tags, right?

However, I’ve seen similar situations to what I described above on blog posts too, where the link from the feed isn’t enough to identify a category, so in these cases it’s not possible to auto-set a category?

[simon]

@renato

just to to be clear: it isn’t possible to use YouTube feeds because they use some custom media tags, right?

Right. I see now that I didn’t answer your initial question:

@renato

the link attributes are the videos themselves, like https://www.youtube.com/watch?v=VID . This means I can’t use these URLs to set the categories by regex, but I could with the source/RSS URLs (each RSS would target a category).

You should be able to get all feeds that have their link attribute set to the www.youtube.com domain to publish to a category by creating a host record with its domain set to www.youtube.com: Something like this:

I don’t think there is a way to set specific categories based on the playlist’s ID though.

[Jonathan5]

@Jonathan5

Is it possible to use any other variables in embed.imported_from apart from %{link}?

Browsing through the forum today I wondered again about this. The reason I ask is that some URLs are long and messy and it would good to “hide” them behind some more useful text in an HTML a tag.

[bekircem]

The feed url section does not appear. I tried hard refresh. what could be the problem?

[simon]

@bekircem

The feed url section does not appear.

Click the “Plugins” link from the admin navigation menu again. You should now see an “RSS Polling” entry in the left-hand sidebar:

It is on its own route, instead of being set via the normal site settings pages. I’m not sure if that’s something that could be changed. It would be more straightforward if the Feed URL could be set as a normal site setting.

[bekircem]

Oh, it seems now. I am sorry for the confusion. Thank you.