I’ve been playing around for this and came up with a solution in Github.
So just a green tick for for the most recent CI job isn’t good enough, as things might have changed in recently, which might break the plugin. Basically you need to re-run the CI jobs when Discourse updates branches.
In this example repository I worked out an efficient solution. It is basically a scheduled workflow which checks the important branches of the Discourse repo so see if there are changes, if there are it will trigger the normal CI workflow which should run the test suite. In you’re readme you can then place some badges to show how does CI workflows ran against the most recent changes.
The monitoring workflow runs in seconds. So when scheduled it would only consume 1 minute of github action time.
Obviously the reliability of this whole setup depends on the plugin/theme-component developer’s effort of creating a good test suite.
And you still have the UX issue that from Discourse’s “update” page you don’t know if the most recent CI job failed for a specific version.
So besides having a monitoring workflow which rebuilds a plugin when the Discourse branch changes, you need to create a build artifact recording the result (pass/fail). In your plugin metadata you should be able to point to this artifact, and discourse should retrieve this artifact to display the compatibility/result in the update interface.
It is not a foolproof construction, but it’s something.