Federation support for Discourse

The issue with ActivityPub is that it is an easy-to-read open standard, but implementing it doesn’t make you “part of the Fediverse” just yet. There’s a bunch of other things to be implemented, and - for a Discussion Forums domain - a custom ActivityPub vocabulary for message exchanges. There are other projects to ‘bootstrap’ from and some libraries to possibly adopt, but there’ll indeed be some significant development that is needed.

In terms of advocacy… I personally feel that, when done well, AP support can add USP’s to the product. Not having to sign up to every forum is one thing… it is a barrier for me too. Shall I sign up to yet one more Discourse, just for this single post I wanna add?

But federation might bring much more value. In my dream scenario I would install a personal Discourse or sign up to an instance, that - like Mastodon - would be completely empty at the start. Then I’d fill it in with community structure myself, based on my needs and interests. Choose this themed group, and add it under this category, add another group.

Update: Cross-posted with @sam :slight_smile: … this was in reaction to @pfaffman

5 Likes

Yes, that’d be a great start. Lemmy link aggregrator works somewhat like that, where it offers a federation of Communities. Note that - though it’d be a very nice-to-have to interact more broadly - federation could be implemented just between Discourse instances / tenants at the start.

Not everything fits to Mastodon… its a Microblogging app, does not support markdown (though other fedi apps do that).

Currently further federated Groups support is being worked on. Lemmy is an example. Bonfire will add Google+ like circles, etc.

3 Likes

Yes! This is the workflow I’d love to see. And promote. :relaxed:

The length of the excerpt would conveniently be configurable, with 0 meaning the entire article rather than an excerpt. Note that the Mastodon 500-character limit is arbitrary, and has nothing to do with the Fediverse, ActivityPub, or ActivityStream. The Mastodon nodes I run currently have 2000-character limits, and the social.kernel.org limit is 31337 characters. Even default Mastodon with its 500-character limit for writing posts still displays longer posts just fine.

A minor difficulty I see is that the user and category namespaces are separate in Discourse but are the same “Actor” in ActivityPub, and at least Mastodon has a fairly restrictive regexp for recognizing Actors. Given @announcements@meta.discourse.org for the #announcements category, this comment would be federated as authored by @mcdanlj@meta.discourse.org

By default, as a Discourse admin I’d want to use the category slug as the Actor name. I suppose that if, when the admin sets up federation for a category, they select the Actor name, which would default to the slug, and would be (like a group name) unique with respect to Discourse User names.

(Incidentally, I used to worry about the Mastodon regex for recognizing actor names, but I think that it’s actually only used for at-mentioning people, which isn’t valuable here anyway. So it might even work to have, say, #documentation:admins represented as @documentation:admins@meta.discourse.org though I’d definitely want to test that with several of the major microblogging systems, definitely including Mastodon and Pleroma. )

3 Likes

I think what this would actually look like from the perspective of, say, a Mastodon or Pleroma user, would be that @announcements@meta.discourse.org would boost / reblog a topic post by, say, @sam@meta.discourse.org and then also boost/reblog a comment post by, say, @mcdanlj@meta.discourse.org as a comment to the OP — Neither the OP nor a comment is actually made by the category, it’s made by a person, in a category.

It might be that the plugin could initially only implement webfinger for the category Actors (in order to be able to follow them), but it might make sense in the end to implement it for individual users as well. I might reasonably, on Mastodon, want to follow @sam@meta.discourse.org to see his posts and comments.

2 Likes

Question: what is the current status of this discussion and the plans for possible implementations? Do you need any testing support for example? Or is that question “far too early” :wink:

Here’s a potentially viable outbound federation support from Discourse idea: Webmentions mediated by Brid.gy.

Flow overview:

  • Topic is created with a link in the first post - to begin with, detect either a single link in the first post, or URL-pasted-in-title
  • Optional: To avoid trivial spam, either wait for the first reply or enable a site setting
  • Probe the page to see if WebMention is supported
  • A WebMention comment is sent with the text “This page is being discussed on =SITE_DOMAIN=: “=TITLE=” =TOPIC_LINK= =CATEGORY_AND_TAGS=” (using site default_locale).
  • The site receives the WebMention (potentially delegating to fed.brid.gy, applying anti spam, etc) and eventually displays that as a comment.

The actor (comment “author”) would take either the high-quality site logo or @system avatar, and the short name of the forum.

5 Likes

Hey, Post OP Here

Just an aside: I didn’t mention it in the post but the protocol supports sending likes and reposts as first class citizens like replies, it would be nice if discourse implemented that too

Pavilion submitted an application to NLnet for 40,000 Euros to add federation support to Discourse (see below). It was rejected and we ended up applying for (and getting accepted) into DAPSI instead.

12 Likes

An update here. Pavilion and CDCK (Discourse.org) are discussing building an ActivityPub plugin for Discourse. After some discussion, we’ve landed on the spec below. Any comments or suggestions welcome before we finalise it. Note that

  • Support for incoming content (e.g. posts on Mastodon etc being imported into Discourse) is intentionally excluded. It will be possible to add this in a later version.

  • Support for following users (as opposed to categories) is also intentionally excluded. It would also be possible to add this in a later version.

  • Substantial parts of the specification are following the approach taken by Lemmy.

  • Pavilion will be building the MVP plugin (I’ll be working on it), and CDCK will own and host it (i.e. it will be a CDCK plugin, not a Pavilion plugin).

Overview

An ActivityPub (AP) Plugin for Discourse.

The goal of the plugin is an MVP implementation of the ActivityPub, ActivityVocab and ActivityStreams specs in Discourse with one demonstrated integration of the MVP functionality for an AP compliant platform, namely Mastodon. As far as possible, the implementation will be built to support further support of the protocols, and any extensions.

This specification concerns enabling AP support on a per-category basis where only the first post in each topic in an AP-enabled category is federated (“first post only”).

Users

The plugin usage will be documented comprehensively on meta in a plugin topic.

Normal User

  1. Subscribes to (aka “follows”) a federation-enabled Discourse category (FDC) on Mastodon (or any other fediverse service) using the category’s handle, e.g. @announcements@meta.discourse.org.

  2. Sees an excerpt of the first post of all FDC topics (posted after they subscribe) in their Mastodon feed, each with a link back to the associated topic, e.g. “Discuss on our forum”.

  3. Any action related to the post in Mastodon does not appear in Discourse.

  4. Any action related to the post in Discourse does not appear in Mastodon.

  5. Federated posts excerpts can be controlled by the post author using markup similar to that used to control topic excerpts, e.g. <div class="note">{text}</div>

Admin

  1. Enables federation on a per-category basis using a category setting. Federation can only be enabled in categories visible to “everyone” on public instances.

  2. Sets the category’s federated username via the category settings UI (cannot be changed).

  3. Sets allow and deny lists of domains via site settings from which activity is automatically accepted or rejected.

  4. Sets a “block list” of federated usernames to be part of a Block object(s) in the server outbox.

  5. Sets the maximum character length of a federated note using a site setting i.e. activitypub_note_excerpt_maxlength.

  6. Sets the text associated with the link included in the first post of a federated Discourse topic in “Customize > Text”.

  7. Sets whether the link back (and text) to the forum is included in federated posts on a per-category basis.

  8. Adds a key pair via site settings to sign federated content with.

Technical

  1. The plugin will include comprehensive testing (unit / integration and js tests).

  2. The category is an automated ActivityPub Actor within Discourse (as a Group ActorType). The federated preferredUsername is set at by the admin when enabling federation and stored in a category custom field. The federated name (aka display name) is the full_name of the category.

  3. Users are Actors in the content federated by the category Actor. The federated preferredUsername is the user’s Discourse username at the time of federation. A user’s preferredUsername is stored in a user custom field in case the user changes their Discourse username. The federated name (aka display name) is the user’s Discourse name.

  4. In general, ActivityPub objects will be associated with their equivalent Discourse objects using custom fields where appropriate (e.g. object or actor ids), and new tables where appropriate (e.g. inbox and outbox).

  5. The primary ActivityPub Activity to implement is Follow and associated activities and models required to implement it, i.e. inbox, outbox and required associated Actions and Collections.

  6. Discourse posts would be federated as ActivityStream Notes, with the content as HTML.

  7. Authentication will be handled using HTTP Signatures, see here and here. All other Security Considerations in the ActivityPub spec will be addressed as appropriate.

  8. Federation endpoints will be protected as appropriate for the specification, i.e. ensure redirect_to_login_if_required is used in controllers, add guardian for category everyone can see permission.

34 Likes

Props for working on such an ambitious project, I’ll follow this with great interest :slight_smile:

7 Likes

I’m really excited to see this news! Thank you! Also, glad you are starting with a simple initial implementation instead of trying to eat the proverbial elephant (not an oblique reference to Mastodon). I hesitate to suggest adding anything, but since you ask for suggestions, one that I hope is insignificantly more work:

What would you think about optionally also posting excerpts of replies, using inReplyTo to thread them? I think this would highlight the dialog occurring on Discourse and be more inviting; along with the “Discuss on our forum” I would expect that seeing the conversation would be more likely to bring more people in.

4 Likes

Thanks for the feedback and support!

The issue with federating replies at this stage is that you may create expectations of engagement which won’t be fulfilled until you have proper support for it. It’s important to keep user expectations clear here. While some users might understand the limitations around those federated replies, others may not. The other issue is the one of trying to reduce the number of technical challenges to tackle at once.

That said, we’ve already contemplated a “whole topic” extension to the “first post only” specification you see above. I’ve shared some of the technical notes for that below to give you a sense of where this could be headed. I would stress that the “whole topic” extension is not part of the initial specification and is just a possibility at this stage.

Whole topic extension
  1. Incoming Notes would be published as new topics or replies (using inReplyTo or Replies).

  2. Discourse security and spam filters will be applied to incoming objects as best as possible using the Accept / Reject objects.

  3. Note posts will be attributed to a staged user associated with the actor id via a custom field. This will require a non-email-based variation of the staged user concept as the user will not have a real email (a placeholder can be generated using the federated actor’s id and preferredUsername).

  4. Associating staged ActivityPub users with standard Discourse users would be possible, but is not currently included in any part of this specification. This is a problem of “federated identity” and it’s associated solutions.

  5. Federated posts would not support federated @mentions, or anything else in this vein outside of the core ActivityPub/Stream specs. Support for such features could be added in later versions, e.g. using Webfinger, following Mastodon.

The key in this first version is to focus on the minimum viable implementation. Once that foundation is laid, full(er) federation support, potentially along the lines of those “whole topic” notes, will be more feasible. I would stress though that the direction of the plugin will be something for CDCK to determine. Anything I mention here are just possible ways the foundation specification could be built on.

6 Likes

Some of the posts and ongoing discussions from the Fediverse:

PS. I would also like to mention that I was approached by Daniël Klabbers core developer of Flarum in November last year who told me they were working on an ActivityPub extension. Don’t know what the status is, but it might be good if all projects in the Forums domain worked together to get as much interoperability going as possible in the space.

Wrt that interop I would also like to point out the Fediverse Enhancement Proposal process, the FEP on Codeberg:

Here e.g. Lemmy recently finalised a FEP for ActivityPub Groups. Discussions take place on the SocialHub at:

(FEP process is picking up steam with the growth of the Fediverse and currently best place to standardize protocol mechanisms)

4 Likes

Thanks @aschrijver, very useful! I’ll personally be quite focused on the MVP implementation spec’ed above, working closely with CDCK. In @mcdanlj’s pithy phrase it’s critical that that effort doesn’t attempt to “swallow the elephant”. We’re keen to hear any constructive feedback on the specification, particularly on the technical front.

I would add though that two other members of Pavilion @nmeyne (a veteran of the SSI community; particularly verifiable credentials), and @merefield are also working in this space, and they’re interested in some of the broader questions you’ve raised.

6 Likes

Great and thanks. I gave a heads-up to Daniël Klabbers via DM pointing to your announcement above.

3 Likes

That makes complete sense to me!

By way of illustration of real misunderstandings from an illusion of fidelity, one of the issues we’ve run into on Maker Forums, where we imported a lot of Google+ communities when Google+ went away, is that the import was sufficiently high fidelity that newcomers to the site don’t realize that they are trying to interact with other users who haven’t (yet) arrived at the site from Google+, and so we have a canned reply for letting them know that the person they are trying to talk to isn’t there to respond to them.

Thanks for providing the details on the “whole topic extension” idea.

5 Likes

In case this is added down the line… fwiw, I really like the approach that Pterotype for Wordpress offered, which is that any federated comments are relayed directly to moderation / group approval before being posted back to the original forum topic as a reply.

In the case of a forum this might be a task better suited to TL4 / TL3 types as opposed to staff.

Personally found this proves especially helpful in preventing off-topic / social media-style replies from overpowering the original discussion topic.

4 Likes

Yes, use of the review queue is something we’ve discussed for incoming content. As you note, this is something to be considered if and when we get there (not in the first version).

Just a note that the administrative matters concerning this work have been sorted, so I’m starting work on the specification next week. Any further technical feedback or notes on the specification are welcome.

It’ll take some months for this to come to fruition, so sit tight.

5 Likes

If a post is deleted in Discourse, can a Delete action be sent to followers?

On Maker Forums, we have enough “google points” that we get plenty of SEO spammers. We have relatively relaxed permissions for new users because a common form of new use is a frustrated person seeking site-relevant help, and it helps us help them faster. But this means that spammers tend to get a spam post in and then it’s relatively quickly flagged as spam, delisted, and then deleted.

I’d love for those federated posts to be deleted and the deletion published via Delete so that we clean up the spam from the federated caches. Is that possibly in scope?

4 Likes

Might be helpful in this case if there were a delay for federating, of some number of hours, to allow spam to be dealt with before federation. (Perhaps even apply the delay only to first posts of relatively new users?)

3 Likes