RSS Polling may take 20 minutes and more to process a feed

zogstrip · September 15, 2026, 8:27am

i have a couple prs in the work that should drastically improve the speed of polling/importing large feeds

github.com/discourse/discourse

PERF: Use an index to look up embeds by URL in `TopicEmbed` (#43643)

main ← perf-topic-embed-url-lookup

merged 08:40AM - 15 Sep 26 UTC

ZogStriP

+58 -4

Previously, `TopicEmbed.topic_embed_by_url` matched `embed_url` with a case-inse…nsitive regex, which can't use an index, so every lookup scanned the whole `topic_embeds` table. RSS polling runs this lookup for every item of every feed on every poll, so re-polling an unchanged 1,600-item feed took ~230s on a site with 100k embeds. This change matches `lower(embed_url)` against the `http` and `https` variants of the normalized URL and adds an index on that expression, bringing the same re-poll down to ~5s. The oldest match is picked in Ruby instead of with `ORDER BY id LIMIT 1` because, until the new index has statistics, Postgres would walk the primary key rather than use it. Re-imports also compare `user_id`s instead of loading the post's user for every item. Reported at https://meta.discourse.org/t/412322

github.com/discourse/discourse

PERF: Use the embed URL index to find imported topics in RSS polling (#43644)

perf-topic-embed-url-lookup ← perf-rss-polling-imported-topics

merged 08:40AM - 15 Sep 26 UTC

ZogStriP

+22 -13

Previously, `ImportedTopics` matched every feed item URL in a single `embed_url …~* ANY(ARRAY[...])` query. Postgres only caches 32 compiled regexes, so with more patterns it recompiles them for every row it checks. With `rss_polling_update_tags` disabled, a 1,600-item feed took ~25 minutes per poll on a site with 100k embeds. This change looks topics up with `TopicEmbed.with_embed_urls`, which uses the `lower(embed_url)` index (~0.1s for the same feed), only loads the columns it needs, and builds lookup keys with `TopicEmbed.embed_url_key` so they can't drift from the core query. Reported at https://meta.discourse.org/t/412322 Stacked on https://github.com/discourse/discourse/pull/43643

Topic		Replies	Views
Are there any upper limits to the RSS Polling plugin? Support rss-polling	6	302	September 13, 2026
Rss polling frequency Support rss-polling	3	110	June 12, 2025
A new admin experience for RSS Polling Announcements rss-polling	5	298	July 24, 2026
RSS Polling Plugin official , rss-polling , included-in-core	4	18185	August 14, 2026
RSS feed not read Support rss-polling	0	90	June 13, 2025

RSS Polling may take 20 minutes and more to process a feed

Related topics