Meer populaire plugins bundelen met Discourse core

[kgrier (Kirk Grier)]

Gezien ik vroeg in mijn Discourse self-hosting ervaring last had van het probleem “plugin failure to rebuild”, kan ik het verlangen waarderen om bekende goede versies in de kern te bundelen. En mogelijk een rijker aanbod aan functies te bieden.

Een klantgerichte organisatie had een peiling kunnen houden over de richting van de kernplugins, of op zijn minst over de timing. Misschien heb ik dat gemist. Als IT-toolprovider voor mijn klanten (d.w.z. eindgebruikers) die echt werk willen verzetten met IT, heb ik veel producten zien evolueren naar overmatige complexiteit en uiteindelijke vervanging. Nu zullen self-hosters plugins die ze niet nodig hebben “rm -fr” en. Ik begrijp dit en sluit me mogelijk bij die club aan. Naar mijn ervaring verhoogt extra code de integratiecomplexiteit, het risico op configuratiefouten en het aanvalsoppervlak. Maar vroeg of laat zal het verwijderen van iets waarvan de app-ontwikkelaar aanneemt dat het er zal zijn, ook iets breken.

Ik had veel liever gezien dat de inspanningen van de Discourse jedi’s gericht waren op het werken aan een robuuste, onderhouden, gescripte methode om cloudopslag van afbeeldingen mogelijk te maken, inclusief CDN-integratie. SMTP-e-mailintegratie is in vergelijking triviaal, en dat is kapotgegaan door wijzigingen bij MailGun en anderen, wat self-hosted sites problemen bezorgde.

[david (David Taylor)]

@kgrier

Now we will have self-hosters “rm -fr” ing plugins they don’t need … But sooner or later removing something that the app developer assumes will be there, will also break something.

Indeed, I would strongly recommend against using rm -rf on these plugins. As you say, there are risks around unexpected inter-dependencies in future. But also, you’ll be creating a diff in the core git repository, which means UI updates via docker_manager are almost certain to fail.

Of course, leaving these plugins in their default ‘disabled’ state is totally supported, and will ensure they don’t have any measurable performance impact.

[elmuerte]

Wat we nodig hebben is een manier om plugins uit te sluiten van detectie, ook al bevinden ze zich in de plugins-directory. Dit lost onderlinge afhankelijkheden niet echt op. Maar het vermindert wel afhankelijkheden van database migraties voor ongebruikte afhankelijkheden (bijv. pgvector voor AI).

[pacharanero (Marcus Baw)]

Voor self-hosters of degenen zoals ik die hostingdiensten voor klanten aanbieden, hier is een eenvoudige grep die zal weergeven of een van de nieuwe gebundelde plugins al in uw containers/app.yml staat.

grep -E 'git clone .*(discourse-(adplugin|affiliate|ai|apple-auth|assign|calendar|chat-integration|data-explorer|gamification|github|graphviz|hcaptcha|login-with-amazon|lti|math|microsoft-auth|oauth2-basic|openid-connect|patreon|policy|post-voting|reactions|rss-polling|solved|subscriptions|templates|topic-voting|user-notes|zendesk-plugin|cakeday))' containers/app.yml

Moet als root worden uitgevoerd, vanuit /var/discourse, en zal alle regels weergeven die handmatig moeten worden verwijderd uit de pluginsectie van de app.yml voordat u opnieuw bouwt.

[mcdanlj (Michael K Johnson)]

@pacharanero Thanks for putting that together!

Might consider changing it to reference containers.*.yml so that it also helps out anyone who has done a standard two-container install where they would instead by in web-only? You really don’t want them in any of your container definitions, after all.

@david would you consider adding that to the top post and maintaining it once you integrate cakeday?

[pacharanero (Marcus Baw)]

@mcdanlj

@pacharanero Thanks for putting that together!

Thank ChatGPT which crafted it exactly correctly in one go from this prompt:

Please scrape the GitHub urls of each of the plugins mentioned in this post
https://meta.discourse.org/t/bundling-more-popular-plugins-with-discourse-core/373574#p-1810533-affected-plugins-3

Compile them into a list and create a unix command which will tell me if any of these plugins are already mentioned in a ‘git clone’ in that Discourse’s containers/app.yml

[mcwumbly (Dave McClure)]

@elmuerte

What we need is a way to exclude plugins from being found even though they are in the plugins directory

On the surface, to me, this does sound like a reasonable thing to consider doing at some point – having a more declarative way to define which plugins are loaded or not at boot time, even if the source is still sitting there on disk.

That said, I don’t know what the feasibility or lift of doing that would be.

[david (David Taylor)]

Het is zeker mogelijk. Maar het voegt complexiteit toe - nog iets om te ondersteunen/onderhouden. En het zou alleen nuttig zijn in single-tenant omgevingen (d.w.z. niet in gedeelde hostingomgevingen, waar verschillende tenants verschillende plugins willen).

Als er al iets is, denk ik dat het nuttiger zou zijn om tijd te besteden aan het verbeteren van de gebruikerservaring en prestaties van ‘uitgeschakelde’ plugins (wat we al zijn begonnen te doen sinds deze grote merge naar de kern).

[Ethsim2 (Ethan)]

ook heb ik niet geprobeerd de plugins te verwijderen, omdat ik het gevoel heb dat dit mijn bugrapporten aan de meta zou compromitteren, net als zelfs de poging tot een gedeelde hostinginstallatie

[MichaIng]

Uff, that was a rough update. Identifying the issue in the mess of rebuild log is not that trivial to start with. Found it, overlooked another plugin to remove from our config two times, hence 3rd rebuild attempt finally passed that part. It would have been really helpful to check for and warn in the first place, that the config needs to be adjusted. discourse-doctor checks (too simple, but can be taken as a start) for plugins in the config, so that can be used as a basis. Probably too late now 3 weeks later, whatever …

But that was not all, we also ran into db:migrate errors. Retried it 2 times, then ran discourse-doctor, which ran the rebuild as well, and weirdly succeeded. I checked its code, and it does absolutely nothing (no change) before the rebuild, and calls the rebuild the exact same way we do. Hence looks like the db:migrate succeeded on 3rd attempt for some reason? I read through the thread that the large number of added plugins add dependencies, which may conflict/be older than what was previously used. Luckily we did not need to add a manual plugin removal step, dependency adjustment or database change, like others seem to have needed. Is it somehow expected that running the db:migrate multiple times will somehow finally succeed? I can just hope nothing is broken …

I fully support this opinion on the matter. Do not suddenly add a whole ton of plugins all at once. If a particular feature is seen as very beneficial for all (starter) instances, discuss it on an individual basis, one by one. It may then be also better to just add it to Discourse natively (with a toggle in settings), instead of keeping it as plugin, bringing their maintainer(s) on board. We are now also considering whether to add a removal step to our config, and what to do with those plugins which are now enabled by default, implying a change to our Discourse instance we did not intend, and which I actually have no time to test and think about now, after midnight local time, also thanks to the upgrade mess and research needed to fix …

Though a question in these regards: Were some features turned into plugins or so? I see narrative bot plugin, which I do not remember from earlier. Its description says “introduces staff”, so at first I thought it would be an addition to discobot, for staff (mods/admins/…) only or so. But from its settings it indeed is discobot, isn’t it? Are all those plugins which appear as enabled now, but were not actively installed via config, features which were present before already?

[sam (Sam Saffron)]

@MichaIng

I see narrative bot plugin, which I do not remember from earlier.

It was installed earlier, we just omitted it from the UI with the rest of the bundled plugins from before. Our UI was improved to properly surface everything that exists. (we had a bunch of omissions including Chat which was hidden via CSS)

---

I have already observed internal dev team velocity increase in the very short time this has been in place. It is leading to a more stable product, something I am very happy about.

No plans to back track here. You need to adjust to the new world.

[SkyeDragon (Skye)]

@Ethsim2

also i haven’t tried to remove the plugins, because i feel it would compromise my Bug reports to the meta, alike even the attempt of a Shared Hosting Installation

IMO if something breaks because a plugin isn’t installed, then that itself is a bug. Core should not depend on plugins. Plugins themselves should clearly list their requirements on their respective pages.

But yes, this is going to make the self-hosted version even less stable going forward, since it’ll be the self-hosters stumbling through these issues. Between this and the forked thread, I really don’t get the impression that stability of self-hosters is a high priority for the team.

[gerhard (Gerhard Schlager)]

@Moin

I think it’s difficult to know which ones have been proofread before and which ones were not.

We hadn’t thought about that in advance, so some approvals might have gone missing. But we managed to transfer a good chunk of them from the plugin projects on Crowdin to core. We’ll do better next time since we now have the tools to transfer approvals between projects.

@Moin

I briefly wondered whether you had checked beforehand to see if there were any open comments that are now missing, since only the texts were added and not all of the data from Crowdin.

No, we didn’t check beforehand, but I checked afterward. None of the plugins had unanswered comments on Crowdin. We have an internal tool that stores all comments that are posted on our Crowdin plugins. We could even use it to post missing comments on Crowdin again, e.g. when Crowdin deletes comments because the source string was updated. It just hasn’t been a priority to implement this.

[merefield (Robert)]

It would really be nice to have one more option here:

“Enabled Plugins”

This would avoid the initial massive list in Installed Plugins.

Also, to facilitate this:

• Allow custom links in the Plugins section (perhaps)
• Make this dropdown respond to a route param filter:

so:

https://example.com/admin/plugins?filter=enabled

[Heliosurge (Dan DeMontmorency)]

@elmuerte

What we need is a way to exclude plugins from being found even though they are in the plugins directory

Agreed. Maybe an option to list enabled core installed plugins. Vs just having show all plugins. Addituonal filtering options definitely better

[Heliosurge (Dan DeMontmorency)]

@SkyeDragon

IMO if something breaks because a plugin isn’t installed, then that itself is a bug. Core should not depend on plugins. Plugins themselves should clearly list their requirements on their respective pages.

I agree with the sentiment. Imho the real disconnect is keeping them listed as plugins.

Once merged it should be moved to something branded like “Core Features” as plugins are seen as optional components that can be installed. Vs part of the main program. So it is a misnomer imho to keep them listed as plugins when the intent is not be uninstalled.

Similar to TC that were merged in core like “personal bubbles” is not listed in Theme Components. Granted in that particular case you cannot disable the former TC. If you wanted to go back to what it was before you needed to create a TC to undo the changesm

[Heliosurge (Dan DeMontmorency)]

@merefield

Enabled Plugins”

This would avoid the initial massive list in Installed Plugins.

Definitely agree with additional filtering options. To also have one for disabled plugins as well.

However after more thought and reading more posts. If the plugins are merged with core. They really maybe should no longer be called plugins and not listed in Plugins. But maybe something like Core Features or Optional Features. Since these are no longer really meant to be able to uninstall them.

[SkyeDragon (Skye)]

@Heliosurge

Once merged it should be moved to something branded like “Core Features” as plugins are seen as optional components that can be installed. Vs part of the main program. So it is a misnomer imho to keep them listed as plugins when the intent is not be uninstalled.

There is no reason properly engineered forum software should require advertising code to run.

If Discourse decides to bake in anti-features, the only thing it’s going to force people to do is either fork Discourse or migrate to something else. Those of us that do not like big tech / ad stuff feel very strongly about this. Discourse will NOT force it on us, no matter how hard it is pushed. (Ubuntu tried this on us and now my highest starred repo is something that removes the ads ;))

[Heliosurge (Dan DeMontmorency)]

@SkyeDragon

There is no reason properly engineered forum software should require advertising code to run.

Not sure I follow. If by advertising code you mean plugins that are merged into the core product vs remaining optional add-ons/installs. If we go back you will find a variety of “Advertising code” was merged in core.

I can see from the DeV team perspective that many of their plugins may have started as plugins to allow more flexible testing before making it integrated in the core program.

I can appreciate like any software there is often features ppl might not use and choose to disable it and up to finding ways to uninstall features.

While I will admit there are a lot of recent merged plugins that I likely would not use. But having a simple disable and filtering them out is something good for all.

I understand that in part as stated by the team the intent is to make things easier with add-ons for self hosted.

Now imho the admin interface should be more customizable like it once was. As this can also help with ppl migrating from another platform by being able to load an admin there that is similar layout as the platform they are coming from. Much like how Linux does this with some mimicking other OSes. But that is another topic.

I can appreciate the feeling that discourse might be starting to head towards bloatware. Reactors demonstrated how much leaner Windows NT could be.