Bundling more popular plugins with Discourse core

Announcements
80

Having been bit by the “plugin failure to rebuild” issue early on in my Discourse self-hosting, I can appreciate the desire to bundle into core known good versions. And potentially offer a richer selection of features.

A customer-focused org could have done a poll on the core plugin direction, or at least the timing. Perhaps I missed that. As a IT tool provider to my customers (i.e. end-users) trying to get real work done with IT, I’ve seen many products rev’d into over complexity and ultimate replacement. Now we will have self-hosters “rm -fr” ing plugins they don’t need. I get this and may join that club. In my experience, additional code increases integration complexity, risk of configuration mistakes, and attack surface. But sooner or later removing something that the app developer assumes will be there, will also break something.

I would have much preferred the Discourse jedi’s efforts gone into working on a robust, maintained, scripted method to enable cloud storage of images including CDN integration. SMTP email integration is trivial by comparison, and that has busted with changes at MailGun and others causing self-hosted sites grief.

81

Indeed, I would strongly recommend against using rm -rf on these plugins. As you say, there are risks around unexpected inter-dependencies in future. But also, you’ll be creating a diff in the core git repository, which means UI updates via docker_manager are almost certain to fail.

Of course, leaving these plugins in their default ‘disabled’ state is totally supported, and will ensure they don’t have any measurable performance impact.

7 Likes
82

What we need is a way to exclude plugins from being found even though they are in the plugins directory. This doesn’t fix really inter-dependencies. But it does reduce DB migration dependencies for unused dependencies (e.g. pgvector for AI).

1 Like
83

For self-hosters or those like me providing hosting services for customers, here’s a simple grep which will list if any of the new bundled plugins are already in your containers/app.yml

grep -E 'git clone .*(discourse-(adplugin|affiliate|ai|apple-auth|assign|calendar|chat-integration|data-explorer|gamification|github|graphviz|hcaptcha|login-with-amazon|lti|math|microsoft-auth|oauth2-basic|openid-connect|patreon|policy|post-voting|reactions|rss-polling|solved|subscriptions|templates|topic-voting|user-notes|zendesk-plugin|cakeday))' containers/app.yml

Needs to run as root, from /var/discourse, and will output any lines that need to be manually removed from the plugins section of the app.yml before you rebuild.

6 Likes
84

@pacharanero Thanks for putting that together!

Might consider changing it to reference containers.*.yml so that it also helps out anyone who has done a standard two-container install where they would instead by in web-only? You really don’t want them in any of your container definitions, after all. :smiling_face:

@david would you consider adding that to the top post and maintaining it once you integrate cakeday?

1 Like
85

Thank ChatGPT which crafted it exactly correctly in one go from this prompt:

Please scrape the GitHub urls of each of the plugins mentioned in this post
https://meta.discourse.org/t/bundling-more-popular-plugins-with-discourse-core/373574#p-1810533-affected-plugins-3

Compile them into a list and create a unix command which will tell me if any of these plugins are already mentioned in a ‘git clone’ in that Discourse’s containers/app.yml

2 Likes
86

On the surface, to me, this does sound like a reasonable thing to consider doing at some point – having a more declarative way to define which plugins are loaded or not at boot time, even if the source is still sitting there on disk.

That said, I don’t know what the feasibility or lift of doing that would be.

1 Like
87

It’s certainly possible. But it adds complexity - yet another thing to support/maintain. And it would only be useful in single-tenant environments (i.e. not in shared hosting environments, where different tenants want different plugins).

If anything, I think it would be more beneficial to spend time improving the UX & performance of ‘disabled’ plugins (which we have already started doing since this big merge into core).

5 Likes
88

also i haven’t tried to remove the plugins, because i feel it would compromise my Bug reports to the meta, alike even the attempt of a Shared Hosting Installation

89

Uff, that was a rough update. Identifying the issue in the mess of rebuild log is not that trivial to start with. Found it, overlooked another plugin to remove from our config two times, hence 3rd rebuild attempt finally passed that part. It would have been really helpful to check for and warn in the first place, that the config needs to be adjusted. discourse-doctor checks (too simple, but can be taken as a start) for plugins in the config, so that can be used as a basis. Probably too late now 3 weeks later, whatever …

But that was not all, we also ran into db:migrate errors. Retried it 2 times, then ran discourse-doctor, which ran the rebuild as well, and weirdly succeeded. I checked its code, and it does absolutely nothing (no change) before the rebuild, and calls the rebuild the exact same way we do. Hence looks like the db:migrate succeeded on 3rd attempt for some reason? I read through the thread that the large number of added plugins add dependencies, which may conflict/be older than what was previously used. Luckily we did not need to add a manual plugin removal step, dependency adjustment or database change, like others seem to have needed. Is it somehow expected that running the db:migrate multiple times will somehow finally succeed? I can just hope nothing is broken …

I fully support this opinion on the matter. Do not suddenly add a whole ton of plugins all at once. If a particular feature is seen as very beneficial for all (starter) instances, discuss it on an individual basis, one by one. It may then be also better to just add it to Discourse natively (with a toggle in settings), instead of keeping it as plugin, bringing their maintainer(s) on board. We are now also considering whether to add a removal step to our config, and what to do with those plugins which are now enabled by default, implying a change to our Discourse instance we did not intend, and which I actually have no time to test and think about now, after midnight local time, also thanks to the upgrade mess and research needed to fix …

Though a question in these regards: Were some features turned into plugins or so? I see narrative bot plugin, which I do not remember from earlier. Its description says “introduces staff”, so at first I thought it would be an addition to discobot, for staff (mods/admins/…) only or so. But from its settings it indeed is discobot, isn’t it? Are all those plugins which appear as enabled now, but were not actively installed via config, features which were present before already?

90

It was installed earlier, we just omitted it from the UI with the rest of the bundled plugins from before. Our UI was improved to properly surface everything that exists. (we had a bunch of omissions including Chat which was hidden via CSS)

I have already observed internal dev team velocity increase in the very short time this has been in place. It is leading to a more stable product, something I am very happy about.

No plans to back track here. You need to adjust to the new world.

91

3 posts were split to a new topic: Help me debug failed migrations