How could a CI workflow for unofficial plugins be implemented?

[thoka (Thomas Kalka)]

I invite you to share thoughts about possible CI workflows, which would run tests on community supplied unofficial plugins.

Some communities depend heavily on plugins, without established maintenance:

@axierr

But we are afraid that during Discourse upgrades something will go wrong due to incompatibilities. Mainly we are concerned about the times with the server down, not so much that certain functionalities do not work or the aesthetics may fail at some point.

@merefield

This is free software so you need to be aware there is no warranty.

NB this is not a slight on Discourse or us. It’s just a technical fact of life because Discourse is evolved with automated tests only on official plugins.

@thoka

I suggest requesting a quote to add tests / specs to the plugin, which warn about problems during update, without much manual testing.

Is anybody aware of other plugins which include tests as best-practice examples?

Since testing plugins depends on a discourse installation as test bed, I wonder how a community supported CI workflow may look like.

Some plugins include specs, like @angus implementation of activitypub, which, as I understand, enable tests for CI.

Currently, I think about two possible ways to improve testing of unofficial plugins, depending on specs / tests, included in the plugin source:

a) to build some machinery, which aides site maintainers to run tests in a staging environment

b) to have some service, which forks a predefined test-image to report tests run on the last published code of the plugin.

What do you think?

Did I miss some already established workflow?

[david (David Taylor)]

This may be useful:

Setup Continuous Integration using GitHub Actions

Overview To build a robust extension for Discourse, it can be wise to include Continuous Integration (CI) into your plugin or theme-component. This will help to detect errors early on and lessen the chances of bugs in your code. Setting up a CI workflow using GitHub Actions to automate builds and testing is an approach the Discourse team uses on all our components, and we recommend you do the same. Setting it up To add automated workflows for GitHub actions to detect, you need to …

[angus (Angus McLeod)]

In addition to CI with backend and frontend tests, which most Pavilion plugins have now, we have a plugin manager system of which CI is one part. This is how it works

[merefield (Robert)]

Yes, and to add to what @angus has shared, you can find our daily compatibility dashboard here:

https://coop.pavilion.tech/plugins?branch=tests-passed

Which displays status of checks for compatibility of Pavilion Plugins (and sometimes others) against tests-passed and stable. This is updated every day, automatically.

This of course relies on tests, including smoke tests, specs (backend) and qunit (front end) tests.

Our subscription plugin (Custom Wizard) has the most test coverage, as you might imagine, but some of our free plugins include a good suite of both back-end and front-end tests (e.g. Locations).

Writing tests is good practice and certainly Pavilion has become yet more disciplined in this space as we have matured as a business.

Critically tests also document your intended functionality, which is extremely important especially during compatibility updates or refactoring.

[thoka (Thomas Kalka)]

@merefield

This is updated every day, automatically.

Impressive. Could you point to the code, which implements this?

[merefield (Robert)]

It’s an architecture as per @angus’ diagram, with multiple repos involved, but this is the status server:

https://github.com/paviliondev/discourse-plugin-manager

It’s also an approach:

• Never implement a fix without checking for a test, and if a suitable test doesn’t exist, add one.
• Preferably develop the test first, show that it fails, then fix the problem and confirm your new test passes.

That way your coverage builds over time …

Moreover:

• Retrofitting tests can be risky as you might misinterpret what the code intended to do … better than doing nothing though …

[pfaffman (Jay Pfaffman)]

All official plugins have specs that you can use as examples. The discourse-plugin-Skelton plugin includes github Actions to run tests at every commit, and I think, daily.

[thoka (Thomas Kalka)]

Do I understand correctly?

@thoka

to build some machinery, which aides site maintainers to run tests in a staging environment

a) This is available via GitHub actions: plugins with proper specs / tests using GitHub actions will have a badge on GitHub, if all tests pass and the status of CI actions is readable by API.

b) Except discourse official plugins and pavilion plugins, there exist no automatic overview for administrators, if used plugins will work in the version intended for update?

Searching for metadata on plugin compatibility, I found Introducing .discourse-compatibility: pinned plugin/theme versions for older Discourse versions via a .discourse-compatibility file.

As I understand, this is a solution to the reverse problem: discourse to old for a plugin.
How to treat the other way around: plugin to old for discourse?

Could /admin/upgrade warn about plugins, failing tests for an intended upgrade?

[merefield (Robert)]

@thoka

b) Except discourse official plugins and pavilion plugins, there exist no automatic overview for administrators, if used plugins will work in the version intended for update?

We originally set out to and provided an unbranded version of our Plugin dashboard where 3rd party authors could put forward their plugin to be included. This didn’t gain any traction, in part because the population of 3rd party plugin developers is very small indeed.

Third party Discourse plugin creation is pretty niche and third party providing plugins with good test coverage is very niche!

A lot of the freelancers in this space have either joined Pavilion or CDCK (or, in time, both!)

So in the end we decide just to collapse our dashboard into our branded community site.