Pavilion submitted an application to NLnet for 40,000 Euros to add federation support to Discourse (see below). It was rejected and we ended up applying for (and getting accepted) into DAPSI instead.
An update here. Pavilion and CDCK (Discourse.org) are discussing building an ActivityPub plugin for Discourse. After some discussion, we’ve landed on the spec below. Any comments or suggestions welcome before we finalise it. Note that
Support for incoming content (e.g. posts on Mastodon etc being imported into Discourse) is intentionally excluded. It will be possible to add this in a later version.
Support for following users (as opposed to categories) is also intentionally excluded. It would also be possible to add this in a later version.
Substantial parts of the specification are following the approach taken by Lemmy.
Pavilion will be building the MVP plugin (I’ll be working on it), and CDCK will own and host it (i.e. it will be a CDCK plugin, not a Pavilion plugin).
An ActivityPub (AP) Plugin for Discourse.
The goal of the plugin is an MVP implementation of the ActivityPub, ActivityVocab and ActivityStreams specs in Discourse with one demonstrated integration of the MVP functionality for an AP compliant platform, namely Mastodon. As far as possible, the implementation will be built to support further support of the protocols, and any extensions.
This specification concerns enabling AP support on a per-category basis where only the first post in each topic in an AP-enabled category is federated (“first post only”).
The plugin usage will be documented comprehensively on meta in a plugin topic.
Subscribes to (aka “follows”) a federation-enabled Discourse category (FDC) on Mastodon (or any other fediverse service) using the category’s handle, e.g.
Sees an excerpt of the first post of all FDC topics (posted after they subscribe) in their Mastodon feed, each with a link back to the associated topic, e.g. “Discuss on our forum”.
Any action related to the post in Mastodon does not appear in Discourse.
Any action related to the post in Discourse does not appear in Mastodon.
Federated posts excerpts can be controlled by the post author using markup similar to that used to control topic excerpts, e.g.
Enables federation on a per-category basis using a category setting. Federation can only be enabled in categories visible to “everyone” on public instances.
Sets the category’s federated username via the category settings UI (cannot be changed).
Sets allow and deny lists of domains via site settings from which activity is automatically accepted or rejected.
Sets a “block list” of federated usernames to be part of a Block object(s) in the server outbox.
Sets the maximum character length of a federated note using a site setting i.e.
Sets the text associated with the link included in the first post of a federated Discourse topic in “Customize > Text”.
Sets whether the link back (and text) to the forum is included in federated posts on a per-category basis.
Adds a key pair via site settings to sign federated content with.
The plugin will include comprehensive testing (unit / integration and js tests).
The category is an automated ActivityPub Actor within Discourse (as a Group ActorType). The federated preferredUsername is set at by the admin when enabling federation and stored in a category custom field. The federated name (aka display name) is the full_name of the category.
Users are Actors in the content federated by the category Actor. The federated
preferredUsernameis the user’s Discourse username at the time of federation. A user’s
preferredUsernameis stored in a user custom field in case the user changes their Discourse username. The federated name (aka display name) is the user’s Discourse name.
In general, ActivityPub objects will be associated with their equivalent Discourse objects using custom fields where appropriate (e.g. object or actor ids), and new tables where appropriate (e.g. inbox and outbox).
The primary ActivityPub Activity to implement is Follow and associated activities and models required to implement it, i.e. inbox, outbox and required associated Actions and Collections.
Discourse posts would be federated as ActivityStream Notes, with the content as HTML.
Authentication will be handled using HTTP Signatures, see here and here. All other Security Considerations in the ActivityPub spec will be addressed as appropriate.
Federation endpoints will be protected as appropriate for the specification, i.e. ensure redirect_to_login_if_required is used in controllers, add guardian for category everyone can see permission.
Props for working on such an ambitious project, I’ll follow this with great interest
I’m really excited to see this news! Thank you! Also, glad you are starting with a simple initial implementation instead of trying to eat the proverbial elephant (not an oblique reference to Mastodon). I hesitate to suggest adding anything, but since you ask for suggestions, one that I hope is insignificantly more work:
What would you think about optionally also posting excerpts of replies, using
inReplyTo to thread them? I think this would highlight the dialog occurring on Discourse and be more inviting; along with the “Discuss on our forum” I would expect that seeing the conversation would be more likely to bring more people in.
Thanks for the feedback and support!
The issue with federating replies at this stage is that you may create expectations of engagement which won’t be fulfilled until you have proper support for it. It’s important to keep user expectations clear here. While some users might understand the limitations around those federated replies, others may not. The other issue is the one of trying to reduce the number of technical challenges to tackle at once.
That said, we’ve already contemplated a “whole topic” extension to the “first post only” specification you see above. I’ve shared some of the technical notes for that below to give you a sense of where this could be headed. I would stress that the “whole topic” extension is not part of the initial specification and is just a possibility at this stage.
Whole topic extension
Incoming Notes would be published as new topics or replies (using inReplyTo or Replies).
Discourse security and spam filters will be applied to incoming objects as best as possible using the Accept / Reject objects.
Note posts will be attributed to a staged user associated with the actor id via a custom field. This will require a non-email-based variation of the staged user concept as the user will not have a real email (a placeholder can be generated using the federated actor’s id and
Associating staged ActivityPub users with standard Discourse users would be possible, but is not currently included in any part of this specification. This is a problem of “federated identity” and it’s associated solutions.
Federated posts would not support federated @mentions, or anything else in this vein outside of the core ActivityPub/Stream specs. Support for such features could be added in later versions, e.g. using Webfinger, following Mastodon.
The key in this first version is to focus on the minimum viable implementation. Once that foundation is laid, full(er) federation support, potentially along the lines of those “whole topic” notes, will be more feasible. I would stress though that the direction of the plugin will be something for CDCK to determine. Anything I mention here are just possible ways the foundation specification could be built on.
Some of the posts and ongoing discussions from the Fediverse:
PS. I would also like to mention that I was approached by Daniël Klabbers core developer of Flarum in November last year who told me they were working on an ActivityPub extension. Don’t know what the status is, but it might be good if all projects in the Forums domain worked together to get as much interoperability going as possible in the space.
Wrt that interop I would also like to point out the Fediverse Enhancement Proposal process, the FEP on Codeberg:
Here e.g. Lemmy recently finalised a FEP for ActivityPub Groups. Discussions take place on the SocialHub at:
(FEP process is picking up steam with the growth of the Fediverse and currently best place to standardize protocol mechanisms)
Thanks @aschrijver, very useful! I’ll personally be quite focused on the MVP implementation spec’ed above, working closely with CDCK. In @mcdanlj’s pithy phrase it’s critical that that effort doesn’t attempt to “swallow the elephant”. We’re keen to hear any constructive feedback on the specification, particularly on the technical front.
I would add though that two other members of Pavilion @nmeyne (a veteran of the SSI community; particularly verifiable credentials), and @merefield are also working in this space, and they’re interested in some of the broader questions you’ve raised.
Great and thanks. I gave a heads-up to Daniël Klabbers via DM pointing to your announcement above.
That makes complete sense to me!
By way of illustration of real misunderstandings from an illusion of fidelity, one of the issues we’ve run into on Maker Forums, where we imported a lot of Google+ communities when Google+ went away, is that the import was sufficiently high fidelity that newcomers to the site don’t realize that they are trying to interact with other users who haven’t (yet) arrived at the site from Google+, and so we have a canned reply for letting them know that the person they are trying to talk to isn’t there to respond to them.
Thanks for providing the details on the “whole topic extension” idea.
In case this is added down the line… fwiw, I really like the approach that Pterotype for Wordpress offered, which is that any federated comments are relayed directly to moderation / group approval before being posted back to the original forum topic as a reply.
In the case of a forum this might be a task better suited to TL4 / TL3 types as opposed to staff.
Personally found this proves especially helpful in preventing off-topic / social media-style replies from overpowering the original discussion topic.
Yes, use of the review queue is something we’ve discussed for incoming content. As you note, this is something to be considered if and when we get there (not in the first version).
Just a note that the administrative matters concerning this work have been sorted, so I’m starting work on the specification next week. Any further technical feedback or notes on the specification are welcome.
It’ll take some months for this to come to fruition, so sit tight.
If a post is deleted in Discourse, can a
Delete action be sent to followers?
On Maker Forums, we have enough “google points” that we get plenty of SEO spammers. We have relatively relaxed permissions for new users because a common form of new use is a frustrated person seeking site-relevant help, and it helps us help them faster. But this means that spammers tend to get a spam post in and then it’s relatively quickly flagged as spam, delisted, and then deleted.
I’d love for those federated posts to be deleted and the deletion published via
Delete so that we clean up the spam from the federated caches. Is that possibly in scope?
Might be helpful in this case if there were a delay for federating, of some number of hours, to allow spam to be dealt with before federation. (Perhaps even apply the delay only to first posts of relatively new users?)
Both useful suggestions, thank you.
I am anticipating we’ll do this in the MVP for the kind of reasons you mention. I’ll discuss it further with the Discourse folks at the right time.
Useful suggestion. Federation will definitely be asynchronous. How the delay is set and handled is something we’ll need to address in some fashion in the MVP.
Specific suggestions like this for the MVP, particularly based on prior use cases, or technical domain expertise, are most welcome.
If it’s not significantly more work up front, also federating out post edits as
Update Activities would be valuable. If it’s one of those things that can be contributed post-MVP, then a design that reasonably accommodates adding that later would be great.
For what it’s worth, I would set the delay low on my instance(s). I would not plan to use it to avoid federating spam posts, because it would reasonably likely be a signal that brings a mod from their fediverse feed into our Discourse to take action. I have
chat_integration_delay_seconds set to 20 for quick typo edits, and would expect to do something similar here. At the very least, that setting sets a precedent for such a delay.
A brief update here! We’re a month into development and, broad brush, we now have the following working
- People can follow categories with ActivityPub enabled
- The first post of topics in ActivityPub-enabled categories are federated to followers
There’s a number of smaller pieces too, but not worth getting into at this stage. There will be at least one more month of development before we start internal testing.
In short, development is going well
That’s huge, thanks for share.
We need freedom in every single aspect that we can build with or without approvement of centralized services.
That’s unstoppable and over-the-counter for every single CEOs, CTOs or wathever-creppy-name decision
And big thanks for every dev involved and the progress they made. I will test when it’s public and share my thoughts.
Cool, might it be applicable to tags as well in that case? Currently we have users subscribe to our tags via rss.
Anyways, congrats! That is very exciting.
That would require tags also to be Actors. Since Users and Categories will be Actors, and “…ActivityPub objects will be associated with their equivalent Discourse objects…” I’m guessing that it would be possible (though I’d guess not part of the MVP since I don’t see it called out in the description) to extend to Actor support for tags, with custom admin configuration per tag.
I’m super interested in running early versions on the SocialHub or another Discourse instance. Let me know how I can be of help.