ICS → Discourse Importer

I’ve built a small utility that continuously syncs events from an iCalendar (ICS) feed into a Discourse category via the REST API.

This isn’t a full Discourse plugin — it runs alongside your Discourse install — so it belongs here in #extras. If you want to display calendar events from an external source (e.g. Google Calendar, University timetable feeds, etc.) inside Discourse topics, this will be useful.

Repository

How it works

  • Reads events from a given ICS feed
  • Matches them against existing topics (by UID or fallback to time/location)
  • Creates or updates topics in your chosen category
  • Can run continuously as a systemd service (safe against duplicate execution via flock)

Requirements

  • Ubuntu 24.04 LTS (tested)
  • Python 3 (already in Ubuntu 24.04 LTS)
  • A Discourse API key
  • A category ID to target for event topics

Example output

Here’s what it looks like when syncing a University timetable ICS feed into Discourse:

Quick start

Clone the repo and install requirements:

git clone https://github.com/Ethsim12/Discourse-ICS-importer-by-REST-API.git /opt/ics-sync
cd /opt/ics-sync
pip install -r requirements.txt

Run a sync once manually:

python3 ics_to_discourse.py \
  --ics "https://example.com/feed.ics" \
  --category-id 4 \
  --site-tz "Europe/London" \
  --static-tags "events,ics"

Set up as a systemd service/timer for continuous sync (example configs in the repo).

2 Likes

the tags were annoying me, so i made sure the search.json look for indexed content of the event - first post of each topic/event

1 Like

Thank you again for the share, this calendar is evolving more and more, getting new features thanks to people like you. I wonder how it will be like in 3-5 years :slight_smile:

1 Like

Brilliant! Thanks for testing it out. Anyone else who wants to try syncing an ICS feed into Discourse, I’d love feedback on whether your feeds behave the same.

2 Likes

A couple of comments.

If I had any time, I’d probably try converting this to a proper plugin. I think it shouldn’t be too hard to create some settings and convert the Python into Ruby and put it in a job.

Another idea, which could be useful for people who are hosted and want to use this, would be to convert the task into a github action and get it to run the task daily. I did this for some scripts a hosted client needed to run daily a while back and it’s working pretty well. It’s at once harder (it requires learning github workflows and how to deal with secrets instead of a good old cron job) and easier (you don’t have to learn how to muck with installing stuff on a machine via a command line interface).

2 Likes

I haven’t tested it lately, but wrapped up the event bbcode parsing in my latest commit to

yes, though it would be nice if the ics_feeds setting were to be broken down, so the admin isn’t inputting a single JSON into UI

1 Like

to be honest i don’t use cron now, i use systemd on a Ubuntu Server 24.04 LTS.

1 Like

this is a luxury that as soon as i have the time i will learn to achieve :wink::face_exhaling:

Not having access to a command line is, IMHO, no luxury at all! :rofl:

1 Like

Haha, to be clear I meant GUI is the real luxury - CLI is the skill I need to work toward.

1 Like

I guess @angus beat you to that by a few years

3 Likes

Behaviour notes from testing ics_to_discourse.py

I’ve been running a series of tests on this script (with and without --time-only-dedupe) and thought it would be useful to document the update/adoption flow in detail.


1. How uniqueness is determined

  • Default mode: adoption requires start + end + location to match exactly.
  • With --time-only-dedupe: adoption requires only start + end; location is treated as “close enough.”

If no existing topic matches these rules, a new topic is created.


2. The role of the UID marker

  • Every event topic gets a hidden HTML marker in the first post:
  <!-- ICSUID:xxxxxxxxxxxxxxxx -->
  • On subsequent runs, the script looks for that marker first.
  • If found, the topic is considered a UID match and updated directly, regardless of how noisy or stale the DESCRIPTION text might be.
  • This makes the UID the true identity key. Visible description fields don’t affect matching.

3. Update flow with UID match

  1. Script fetches the first post and strips the marker:
old_clean = strip_marker(old_raw)
fresh_clean = strip_marker(fresh_raw)
  1. If old_clean == fresh_clean: no update (avoids churn).
  2. If they differ: check whether the change is “meaningful”:
meaningful = (
    _norm_time(old_attrs.get("start")) != _norm_time(new_attrs.get("start"))
    or _norm_time(old_attrs.get("end")) != _norm_time(new_attrs.get("end"))
    or _norm_loc(old_attrs.get("location")) != _norm_loc(new_attrs.get("location"))
)
  • If meaningful = True → update with bump (topic rises in Latest).

  • If meaningful = False → update quietly (bypass_bump=True → revision only, no bump).

    1. Tags are merged (ensures static/default tags are present, never removes moderator/manual ones).
    2. Title and category are never changed on update.

  1. Update flow with no UID match
    1. Script attempts adoption:
      • Builds candidate triples of start/end/location (or start/end only with --time-only-dedupe).
      • Searches /search.json and /latest.json for an existing event with matching attributes.
      • If found → adopt that topic, retrofit UID marker + tags (body left unchanged at this stage).
      • If not found → create a brand new topic with the marker and tags.
    2. Once adopted or created, all future syncs will resolve directly by UID.

  1. Practical consequences
    • Time changes
    • Default: adoption fails (times differ) → new topic created.
    • With --time-only-dedupe: adoption fails the same way; new topic created.
    • Location changes
    • Default: adoption fails (location differs) → new topic created.
    • With --time-only-dedupe: adoption succeeds (times match), but location difference is flagged as “meaningful” → update with bump.
    • Description changes
    • If DESCRIPTION text changes but start/end/location do not:
    • Body is updated quietly (bypass_bump=True).
    • Topic revision created, but no bump in Latest.
    • If DESCRIPTION is unchanged (or only noise such as Last Updated: that normalizes away), no update occurs at all.
    • UID marker
    • Ensures reliable matching on future syncs.
    • Means noisy DESCRIPTION fields don’t affect whether the correct topic is found.

  1. Why the DESCRIPTION sometimes “stays the same”

The script compares the entire body (minus the UID marker).
If only a volatile line like Last Updated: is different, but it normalizes away (e.g. whitespace, line endings, Unicode), old_clean and fresh_clean appear identical → no update is made.
This is by design, to prevent churn from feed noise.


Summary

  • Time defines uniqueness (always creates new topic when times change).
  • Location changes → visible bump (so users notice venue updates).
  • Description changes → quiet update (revision but no bump).
  • UID marker = reliable identity key, ensures the correct topic is always found, even if DESCRIPTION is stale or noisy.

This strikes a good balance: important changes surface in Latest, unimportant churn stays invisible.

Looking back, it’s kind of hilarious how this whole saga unfolded.
The importer script itself is now rock-solid: UID markers, dedupe logic, meaningful vs. quiet updates, tag namespaces… all the stuff you’d actually want in production. The behaviours line up perfectly with the notes i posted — times define uniqueness, locations trigger a bump, descriptions update quietly, and UID markers keep everything anchored. It’s elegant, it’s predictable, it’s done. :white_check_mark:

Meanwhile, the poor Meta topic that hosted it all was… well, doomed.
It began life replying as a sockpuppet (strong start :socks:), ballooned into a Frankenstein thread of code dumps and screenshots, then evolved into a pseudo-changelog with more commits than the repo itself. And just as the script finally became stable? Scheduled for deletion. :skull:

Honestly, it’s poetic. The script’s entire purpose is to stop duplicate events from cluttering up your forum. The topic itself? Seen as a duplicate, quietly marked for garbage collection. The very fate it was built to prevent became its destiny. :wastebasket:

So here’s to the doomed topic:
You didn’t bump Latest, but you bumped our hearts. :heart:

2 Likes

How did you get on with moving it to a Discourse plugin? Or better yet, as a PR on the existing Discourse Calendar (and Event) Plugin?

I’m reluctant to jump into the config and maintenance required to run your awesome looking script as is (and suspect that many self-hosters would be in the same boat).

How is this script better than the plugin? (Oh, maybe you can’t install plugins?) If the plugin doesn’t do what’s required, maybe submit a PR?

Thanks for the nudge!

Quick status: I’m currently running three instances of my Python ICS→Discourse importer (Uni timetable, Sports Centre bookings, and an Outlook calendar). I did start wrapping it as a Discourse plugin, but the plugin version fell short of the script’s feature-set — mainly because each feed needs bespoke handling (UID quirks, partial updates, cancellations, noisy revisions, etc.). Angus’s plugin is great for many cases; my use cases seem more “feed-specific”.

I also have an open PR against core aimed at reducing the “Latest” blue button noise during large/bursty ICS updates. With busy feeds (like university timetables) a batch of low-value edits can keep “Latest” bouncing; the PR effectively no-ops the “New Topics” button when Latest has sat open while an automated batch runs. Happy to cross-link that PR here if useful.

Longer term: I’m on self-hosted IONOS right now. If I move to official hosting later, I’d still love a way to keep the Python flow (or an equivalent) without needing Enterprise features, if ICS inbound exists there. I suspect a generic core/plugin solution could work if it allowed pluggable “adapters” per feed while keeping strong idempotency (ICS UID), cancellation handling, and edit-without-bump semantics.

If there’s interest, I can sketch a minimal adapter interface and a migration path from my Python script to a Ruby job, or contribute feed-agnostic pieces (UID mapping, debounce/no-bump updates, cancellation logic) to the calendar/events plugin.

1 Like

That’s a good question, Nathan — and I think there’s definitely space for a minimal, feed-agnostic approach that could live either as a small extension to the Calendar/Event plugin or as a lightweight core job.

For a PR to be generally useful, the key seems to be making the importer adapter-based rather than feed-specific. Something like:

  • Each feed defines a small adapter (could be Python, YAML, or Ruby) that maps ICS fields → Discourse topic fields (title, body, tags, start, end, location, etc.).
  • Core handles idempotency (UID ↔ topic ID mapping), cancellation (STATUS:CANCELLED), and quiet edits (update without bumping Latest).
  • Plugins or site settings could configure polling interval, tag mappings, and bump policy (always, never, on major change).

That way, institutions with noisy or complex feeds (university timetables, room bookings, Outlook calendars, etc.) can provide an adapter suited to their data without hardcoding anything in core.

If there’s interest, I’d be happy to outline that adapter interface or prototype the core “ICS upsert” helper as a Ruby job that others can build on — so that this can gradually evolve from standalone Python scripts to something maintainable and generic within Discourse’s ecosystem.

2 Likes