I invite you to share thoughts about possible CI workflows, which would run tests on community supplied unofficial plugins. Some communities depend heavily on plugins, without established maintenance: [image]Unofficial plugins management during updates But we are afraid that during Discourse upgrades something will go wrong due to incompatibilities. Mainly we are concerned about the times with the server down, not so much that certain functionalities do not work or the aesthetics may fail at some point. [image]Unofficial plugins management during updates This is free software so you need to be aware there is no warranty. NB this is not a slight on Discourse or us. It’s just a technical fact of life because Discourse is evolved with automated tests only on official plugins. [image][PAID] upgrade MLM-daily-Summary plugin to work with discourse 3 and higher I suggest requesting a quote to add tests / specs to the plugin, which warn about problems during update, without much manual testing. Is anybody aware of other plugins which include tests as best-practice examples? Since testing plugins depends on a discourse installation as test bed, I wonder how a community supported CI workflow may look like. Some plugins include specs, like @angus implementation of activitypub, which, as I understand, enable tests for CI. Currently, I think about two possible ways to improve testing of unofficial plugins, depending on specs / tests, included in the plugin source: a) to build some machinery, which aides site maintainers to run tests in a staging environment b) to have some service, which forks a predefined test-image to report tests run on the last published code of the plugin. What do you think? Did I miss some already established workflow?

This may be useful: [image] Setup Continuous Integration using GitHub Actions Developer Guides mag Overview To build a robust extension for Discourse, it can be wise to include Continuous Integration (CI) into your plugin or theme-component. This will help to detect errors early on and lessen the chances of bugs in your code. Setting up a CI workflow using GitHub Actions to automate builds and testing is an approach the Discourse team uses on all our components, and we recommend you do the same. gear Setting it up To add automated workflows for GitHub actions to detect, you need to …

In addition to CI with backend and frontend tests, which most Pavilion plugins have now, we have a plugin manager system of which CI is one part. This is how it works [image]

Yes, and to add to what @angus has shared, you can find our daily compatibility dashboard here: https://coop.pavilion.tech/plugins?branch=tests-passed Which displays status of checks for compatibility of Pavilion Plugins (and sometimes others) against tests-passed and stable. This is updated every day, automatically. This of course relies on tests, including smoke tests, specs (backend) and qunit (front end) tests. Our subscription plugin (Custom Wizard) has the most test coverage, as you might imagine, but some of our free plugins include a good suite of both back-end and front-end tests (e.g. Locations). Writing tests is good practice and certainly Pavilion has become yet more disciplined in this space as we have matured as a business. Critically tests also document your intended functionality, which is extremely important especially during compatibility updates or refactoring.

[image] Robert: This is updated every day, automatically. Impressive. Could you point to the code, which implements this?

It’s an architecture as per @angus’ diagram, with multiple repos involved, but this is the status server: https://github.com/paviliondev/discourse-plugin-manager It’s also an approach: Never implement a fix without checking for a test, and if a suitable test doesn’t exist, add one. Preferably develop the test first, show that it fails, then fix the problem and confirm your new test passes. That way your coverage builds over time … Moreover: Retrofitting tests can be risky as you might misinterpret what the code intended to do … better than doing nothing though …

Do I understand correctly? [image] Thomas Kalka: to build some machinery, which aides site maintainers to run tests in a staging environment a) This is available via GitHub actions: plugins with proper specs / tests using GitHub actions will have a badge on GitHub, if all tests pass and the status of CI actions is readable by API. b) Except discourse official plugins and pavilion plugins, there exist no automatic overview for administrators, if used plugins will work in the version intended for update? Searching for metadata on plugin compatibility, I found Introducing .discourse-compatibility: pinned plugin/theme versions for older Discourse versions via a .discourse-compatibility file. As I understand, this is a solution to the reverse problem: discourse to old for a plugin. How to treat the other way around: plugin to old for discourse? Could /admin/upgrade warn about plugins, failing tests for an intended upgrade?

[image] Thomas Kalka: b) Except discourse official plugins and pavilion plugins, there exist no automatic overview for administrators, if used plugins will work in the version intended for update? We originally set out to and provided an unbranded version of our Plugin dashboard where 3rd party authors could put forward their plugin to be included. This didn’t gain any traction, in part because the population of 3rd party plugin developers is very small indeed. Third party Discourse plugin creation is pretty niche and third party providing plugins with good test coverage is very niche! :sweat_smile: A lot of the freelancers in this space have either joined Pavilion or CDCK (or, in time, both!) So in the end we decide just to collapse our dashboard into our branded community site.

How could a CI workflow for unofficial plugins be implemented?

Development

pfaffman (Jay Pfaffman) July 16, 2023, 9:35am 7

All official plugins have specs that you can use as examples. The discourse-plugin-Skelton plugin includes github Actions to run tests at every commit, and I think, daily.

Topic		Replies	Views
Setup Continuous Integration using GitHub Actions Developer Guides how-to , code	8	1836	March 3, 2025
Smoke-testing plugins during upgrade process Feature	22	2954	May 15, 2025
Give Plugin versions for minimum and maximum discourse version Feature	5	304	September 5, 2025
Unofficial plugins management during updates Self-hosting	5	1320	October 22, 2020
Suggestions for better plugin maintenance Development	15	3211	November 7, 2016

How could a CI workflow for unofficial plugins be implemented?

Related topics