Federation support for Discourse

There is huge potential here, for sure. The possibilities and feasibilities warrant thorough and sober consideration.

This might be Xanadu territory… (Read Jeff’s post above and follow the link in it.)   Such broad integration is also probably undesirable to most admin/operators of Discourse instances. A formal survey of admins & operators is in order (as opposed to “survey by traffic level” in discussion forums).

This looks like an invitation to security hacks since it dangles the “keys to the kingdom” of anyone who can hack the user credentials of any one of the federated sites. At the very least these features would have to be turned off by default or explicitly loaded as plugins–with most security features enabled and locked down. Zoom taught us this lesson by rightfully leaving their platform open and easy to use (to quickly gain and cultivate users on ramp-up) but then had to quickly lock down once the ruffians found the front door unlocked.

Nevertheless, a micro-federation of sites would be a boost to Discourse implementation. If I could create a circle of sites for my municipality that unites the same pool of users (county/city citizens for example), this would put people in communication and could enable some positive outcomes in local governance and community life. This same principle also applies to any business large enough to justify the overhead of administering multiple Discourse instances so that each division can have its own Discourse container with easy navigations to other divisions. THAT would be the embodiment of meta.discourse.

Jeff, Sam, and Co. [@codinghorror @sam] and/or their Steering Committee would first need to decide, however, whether Discourse is a social platform or an enterprise platform. My vote is for enterprise because I see the most potential there for both sides of this split. Enterprise will produce the greatest financial reward and immediate social benefit by improving businesses’ ability to support employees (take good care of the business and the business can take good care of you). Some of those commercial funds could probably then support a social.discourse.org foundation. It’s also very likely that features useful to an enterprise will carry over well to the social domain. These two factors make for an overall win-win.

The two domains need to be distinct, though, because they’re so different. And nice-to-have features will necessarily need to favor whichever version is the primary use case.

Happily, the benefits flow both ways since anyone interested in social.discourse.org gets their reward from the social aspects of community building and being able to pursue community-related activities, so will work–and often work hard–for these non-financial rewards. This social.discourse.org work will inevitably lead to development and features that are useful in the enterprise context and thereby return value to Enterprise Discourse Incorporated in exchange for non-profit support of the Social Discourse Foundation. Even more win-win.

Notice that there isn’t a single exclamation point above. These are just plain facts and statements of likely outcomes. Very pragmatic.

I’ve been looking for a suitable GroupWare platform for my businesses for several years now. Slack briefly inspired high hopes since it was developed for internal business use (and has a very interesting origin story) but didn’t even make it through first-round screening. I’m very impressed with Discourse and optimistic about it.

1 Like

That’s the whole point of this topic, i.e. between Discourse instances.

It is stated in the OP:

2 Likes

Right, and

and

a micro-federation of sites would be a boost

All on-topic, no? :smiley:

Not necessarily a prerequisite. There is the plugin ecosystem to be considered.

Significantly large features are often developed as plugins or even Theme Components (where possible) that do not involve such an administrative overhead nor centralised planning.

That’s part of the beauty of the Discourse ecosystem.

Pavilion, for example, created Locations, Topic List Previews, Multilingual, Follow, Layouts, Custom Wizard (to name but a few) all without consultation with CDCK. Hence, in part, our involvement in this discussion.

7 Likes

God bless our plug-in developers! They generously provide proof-of-concept examples that can be field-tested and considered for inclusion in the core product. Your multilingual plug-in is an excellent example!

Over at python.org, this role is played by library developers who sometimes create “gotta have that” functions that are accepted for inclusion in the Python core package or standard distribution library.

2 Likes

I have been advocating Discourse to add federation support in a bunch of topics on this forum, like here. With the Fediverse going mainstream now, and Tumblr and maybe Flickr and such adding federation support, the question is if Discourse has more interest for adding support as well.

I was contacted by the core developer of Flarum for advice on AP implementation and passed some links back, among which the one just mentioned.

PS. At SocialHub discourse forum, developer community for the Fediverse, we are looking for a long time how we can be part of Fediverse as separate forums have shown to be too much of a barrier for most people to participate.

7 Likes

I’ve gotten mildly interested in Mastodon lately (enough to purchase a domain name, but not actually install Mastodon), so I’ve read a bit about authenticating Discourse with Mastodon.

I found a few posts suggesting that it’s harder than people thought. IIRC, it looked like a grant had been offered to develop a plugin but it seemed to have fallen though. My wild guess is that it’s a $10,000 problem; if I’m right, that’s the kind of advocacy you’re going to need.

Edit: but I was conflating authentication with federation.

1 Like

My general concern here is that the ideas are generally enormous, it is super hard to break them down into small pieces.

I do find the idea of federating interesting. A possible concrete implementation could be:

  • Allow Discourse admins to federate a category.
  • Users on mastodon can then follow it, eg follow: announcements@meta.discourse.org
  • When new announcement topics are created, a new post is federated with an excerpts and link to the original.
  • As users respond on discourse, new replies are federated and attributed (as replies to original topic)
  • When someone on the fediverse replies, a “shadow” post is made on the topic attributed to the poster on mastodon.

Something like that is at least small enough to properly fit in my head.

13 Likes

The issue with ActivityPub is that it is an easy-to-read open standard, but implementing it doesn’t make you “part of the Fediverse” just yet. There’s a bunch of other things to be implemented, and - for a Discussion Forums domain - a custom ActivityPub vocabulary for message exchanges. There are other projects to ‘bootstrap’ from and some libraries to possibly adopt, but there’ll indeed be some significant development that is needed.

In terms of advocacy… I personally feel that, when done well, AP support can add USP’s to the product. Not having to sign up to every forum is one thing… it is a barrier for me too. Shall I sign up to yet one more Discourse, just for this single post I wanna add?

But federation might bring much more value. In my dream scenario I would install a personal Discourse or sign up to an instance, that - like Mastodon - would be completely empty at the start. Then I’d fill it in with community structure myself, based on my needs and interests. Choose this themed group, and add it under this category, add another group.

Update: Cross-posted with @sam :slight_smile: … this was in reaction to @pfaffman

5 Likes

Yes, that’d be a great start. Lemmy link aggregrator works somewhat like that, where it offers a federation of Communities. Note that - though it’d be a very nice-to-have to interact more broadly - federation could be implemented just between Discourse instances / tenants at the start.

Not everything fits to Mastodon… its a Microblogging app, does not support markdown (though other fedi apps do that).

Currently further federated Groups support is being worked on. Lemmy is an example. Bonfire will add Google+ like circles, etc.

3 Likes

Yes! This is the workflow I’d love to see. And promote. :relaxed:

The length of the excerpt would conveniently be configurable, with 0 meaning the entire article rather than an excerpt. Note that the Mastodon 500-character limit is arbitrary, and has nothing to do with the Fediverse, ActivityPub, or ActivityStream. The Mastodon nodes I run currently have 2000-character limits, and the social.kernel.org limit is 31337 characters. Even default Mastodon with its 500-character limit for writing posts still displays longer posts just fine.

A minor difficulty I see is that the user and category namespaces are separate in Discourse but are the same “Actor” in ActivityPub, and at least Mastodon has a fairly restrictive regexp for recognizing Actors. Given @announcements@meta.discourse.org for the #announcements category, this comment would be federated as authored by @mcdanlj@meta.discourse.org

By default, as a Discourse admin I’d want to use the category slug as the Actor name. I suppose that if, when the admin sets up federation for a category, they select the Actor name, which would default to the slug, and would be (like a group name) unique with respect to Discourse User names.

(Incidentally, I used to worry about the Mastodon regex for recognizing actor names, but I think that it’s actually only used for at-mentioning people, which isn’t valuable here anyway. So it might even work to have, say, #documentation:admins represented as @documentation:admins@meta.discourse.org though I’d definitely want to test that with several of the major microblogging systems, definitely including Mastodon and Pleroma. )

3 Likes

I think what this would actually look like from the perspective of, say, a Mastodon or Pleroma user, would be that @announcements@meta.discourse.org would boost / reblog a topic post by, say, @sam@meta.discourse.org and then also boost/reblog a comment post by, say, @mcdanlj@meta.discourse.org as a comment to the OP — Neither the OP nor a comment is actually made by the category, it’s made by a person, in a category.

It might be that the plugin could initially only implement webfinger for the category Actors (in order to be able to follow them), but it might make sense in the end to implement it for individual users as well. I might reasonably, on Mastodon, want to follow @sam@meta.discourse.org to see his posts and comments.

2 Likes

Question: what is the current status of this discussion and the plans for possible implementations? Do you need any testing support for example? Or is that question “far too early” :wink:

Here’s a potentially viable outbound federation support from Discourse idea: Webmentions mediated by Brid.gy.

Flow overview:

  • Topic is created with a link in the first post - to begin with, detect either a single link in the first post, or URL-pasted-in-title
  • Optional: To avoid trivial spam, either wait for the first reply or enable a site setting
  • Probe the page to see if WebMention is supported
  • A WebMention comment is sent with the text “This page is being discussed on =SITE_DOMAIN=: “=TITLE=” =TOPIC_LINK= =CATEGORY_AND_TAGS=” (using site default_locale).
  • The site receives the WebMention (potentially delegating to fed.brid.gy, applying anti spam, etc) and eventually displays that as a comment.

The actor (comment “author”) would take either the high-quality site logo or @system avatar, and the short name of the forum.

5 Likes

Hey, Post OP Here

Just an aside: I didn’t mention it in the post but the protocol supports sending likes and reposts as first class citizens like replies, it would be nice if discourse implemented that too

Pavilion submitted an application to NLnet for 40,000 Euros to add federation support to Discourse (see below). It was rejected and we ended up applying for (and getting accepted) into DAPSI instead.

12 Likes

An update here. Pavilion and CDCK (Discourse.org) are discussing building an ActivityPub plugin for Discourse. After some discussion, we’ve landed on the spec below. Any comments or suggestions welcome before we finalise it. Note that

  • Support for incoming content (e.g. posts on Mastodon etc being imported into Discourse) is intentionally excluded. It will be possible to add this in a later version.

  • Support for following users (as opposed to categories) is also intentionally excluded. It would also be possible to add this in a later version.

  • Substantial parts of the specification are following the approach taken by Lemmy.

  • Pavilion will be building the MVP plugin (I’ll be working on it), and CDCK will own and host it (i.e. it will be a CDCK plugin, not a Pavilion plugin).

Overview

An ActivityPub (AP) Plugin for Discourse.

The goal of the plugin is an MVP implementation of the ActivityPub, ActivityVocab and ActivityStreams specs in Discourse with one demonstrated integration of the MVP functionality for an AP compliant platform, namely Mastodon. As far as possible, the implementation will be built to support further support of the protocols, and any extensions.

This specification concerns enabling AP support on a per-category basis where only the first post in each topic in an AP-enabled category is federated (“first post only”).

Users

The plugin usage will be documented comprehensively on meta in a plugin topic.

Normal User

  1. Subscribes to (aka “follows”) a federation-enabled Discourse category (FDC) on Mastodon (or any other fediverse service) using the category’s handle, e.g. @announcements@meta.discourse.org.

  2. Sees an excerpt of the first post of all FDC topics (posted after they subscribe) in their Mastodon feed, each with a link back to the associated topic, e.g. “Discuss on our forum”.

  3. Any action related to the post in Mastodon does not appear in Discourse.

  4. Any action related to the post in Discourse does not appear in Mastodon.

  5. Federated posts excerpts can be controlled by the post author using markup similar to that used to control topic excerpts, e.g. <div class="note">{text}</div>

Admin

  1. Enables federation on a per-category basis using a category setting. Federation can only be enabled in categories visible to “everyone” on public instances.

  2. Sets the category’s federated username via the category settings UI (cannot be changed).

  3. Sets allow and deny lists of domains via site settings from which activity is automatically accepted or rejected.

  4. Sets a “block list” of federated usernames to be part of a Block object(s) in the server outbox.

  5. Sets the maximum character length of a federated note using a site setting i.e. activitypub_note_excerpt_maxlength.

  6. Sets the text associated with the link included in the first post of a federated Discourse topic in “Customize > Text”.

  7. Sets whether the link back (and text) to the forum is included in federated posts on a per-category basis.

  8. Adds a key pair via site settings to sign federated content with.

Technical

  1. The plugin will include comprehensive testing (unit / integration and js tests).

  2. The category is an automated ActivityPub Actor within Discourse (as a Group ActorType). The federated preferredUsername is set at by the admin when enabling federation and stored in a category custom field. The federated name (aka display name) is the full_name of the category.

  3. Users are Actors in the content federated by the category Actor. The federated preferredUsername is the user’s Discourse username at the time of federation. A user’s preferredUsername is stored in a user custom field in case the user changes their Discourse username. The federated name (aka display name) is the user’s Discourse name.

  4. In general, ActivityPub objects will be associated with their equivalent Discourse objects using custom fields where appropriate (e.g. object or actor ids), and new tables where appropriate (e.g. inbox and outbox).

  5. The primary ActivityPub Activity to implement is Follow and associated activities and models required to implement it, i.e. inbox, outbox and required associated Actions and Collections.

  6. Discourse posts would be federated as ActivityStream Notes, with the content as HTML.

  7. Authentication will be handled using HTTP Signatures, see here and here. All other Security Considerations in the ActivityPub spec will be addressed as appropriate.

  8. Federation endpoints will be protected as appropriate for the specification, i.e. ensure redirect_to_login_if_required is used in controllers, add guardian for category everyone can see permission.

34 Likes

Props for working on such an ambitious project, I’ll follow this with great interest :slight_smile:

7 Likes

I’m really excited to see this news! Thank you! Also, glad you are starting with a simple initial implementation instead of trying to eat the proverbial elephant (not an oblique reference to Mastodon). I hesitate to suggest adding anything, but since you ask for suggestions, one that I hope is insignificantly more work:

What would you think about optionally also posting excerpts of replies, using inReplyTo to thread them? I think this would highlight the dialog occurring on Discourse and be more inviting; along with the “Discuss on our forum” I would expect that seeing the conversation would be more likely to bring more people in.

4 Likes

Thanks for the feedback and support!

The issue with federating replies at this stage is that you may create expectations of engagement which won’t be fulfilled until you have proper support for it. It’s important to keep user expectations clear here. While some users might understand the limitations around those federated replies, others may not. The other issue is the one of trying to reduce the number of technical challenges to tackle at once.

That said, we’ve already contemplated a “whole topic” extension to the “first post only” specification you see above. I’ve shared some of the technical notes for that below to give you a sense of where this could be headed. I would stress that the “whole topic” extension is not part of the initial specification and is just a possibility at this stage.

Whole topic extension
  1. Incoming Notes would be published as new topics or replies (using inReplyTo or Replies).

  2. Discourse security and spam filters will be applied to incoming objects as best as possible using the Accept / Reject objects.

  3. Note posts will be attributed to a staged user associated with the actor id via a custom field. This will require a non-email-based variation of the staged user concept as the user will not have a real email (a placeholder can be generated using the federated actor’s id and preferredUsername).

  4. Associating staged ActivityPub users with standard Discourse users would be possible, but is not currently included in any part of this specification. This is a problem of “federated identity” and it’s associated solutions.

  5. Federated posts would not support federated @mentions, or anything else in this vein outside of the core ActivityPub/Stream specs. Support for such features could be added in later versions, e.g. using Webfinger, following Mastodon.

The key in this first version is to focus on the minimum viable implementation. Once that foundation is laid, full(er) federation support, potentially along the lines of those “whole topic” notes, will be more feasible. I would stress though that the direction of the plugin will be something for CDCK to determine. Anything I mention here are just possible ways the foundation specification could be built on.

6 Likes