RSS 轮询处理源可能需要 20 分钟甚至更长时间

zogstrip · 2026 年9 月 15 日 08:27

我有几个正在开发中的 PR，应该能大幅提升轮询/导入大型 feed 的速度

github.com/discourse/discourse

PERF: Use an index to look up embeds by URL in `TopicEmbed` (#43643)

main ← perf-topic-embed-url-lookup

merged 08:40AM - 15 Sep 26 UTC

ZogStriP

+58 -4

Previously, `TopicEmbed.topic_embed_by_url` matched `embed_url` with a case-inse…nsitive regex, which can't use an index, so every lookup scanned the whole `topic_embeds` table. RSS polling runs this lookup for every item of every feed on every poll, so re-polling an unchanged 1,600-item feed took ~230s on a site with 100k embeds. This change matches `lower(embed_url)` against the `http` and `https` variants of the normalized URL and adds an index on that expression, bringing the same re-poll down to ~5s. The oldest match is picked in Ruby instead of with `ORDER BY id LIMIT 1` because, until the new index has statistics, Postgres would walk the primary key rather than use it. Re-imports also compare `user_id`s instead of loading the post's user for every item. Reported at https://meta.discourse.org/t/412322

github.com/discourse/discourse

PERF: Use the embed URL index to find imported topics in RSS polling (#43644)

perf-topic-embed-url-lookup ← perf-rss-polling-imported-topics

merged 08:40AM - 15 Sep 26 UTC

ZogStriP

+22 -13

Previously, `ImportedTopics` matched every feed item URL in a single `embed_url …~* ANY(ARRAY[...])` query. Postgres only caches 32 compiled regexes, so with more patterns it recompiles them for every row it checks. With `rss_polling_update_tags` disabled, a 1,600-item feed took ~25 minutes per poll on a site with 100k embeds. This change looks topics up with `TopicEmbed.with_embed_urls`, which uses the `lower(embed_url)` index (~0.1s for the same feed), only loads the columns it needs, and builds lookup keys with `TopicEmbed.embed_url_key` so they can't drift from the core query. Reported at https://meta.discourse.org/t/412322 Stacked on https://github.com/discourse/discourse/pull/43643

话题		回复	浏览量
Are there any upper limits to the RSS Polling plugin? Support rss-polling	6	302	2026 年9 月 13 日
Rss polling frequency Support rss-polling	3	110	2025 年6 月 12 日
A new admin experience for RSS Polling Announcements rss-polling	5	297	2026 年7 月 24 日
RSS Polling Plugin official , rss-polling , included-in-core	4	18185	2026 年8 月 14 日
RSS feed not read Support rss-polling	0	90	2025 年6 月 13 日

RSS 轮询处理源可能需要 20 分钟甚至更长时间

相关话题