Setting up plugin continuous integration tests on Travis CI

(David Taylor) #1

Following on from Beginner’s Guide to Creating Discourse Plugins Part 6: Acceptance Tests

To set up continuous integration for your plugin, you can use our discourse-plugin-ci repository. This will run the Discourse core linters (prettify, eslint & rubocop), your plugin’s Rspec tests, and your plugin’s Qunit tests.

For an example, check out discourse-chat-integration

Setting it up

  • Copy & paste .travis.yml from here to the root directory of your plugin’s repository.

  • Head over to, and sign up with your GitHub account

  • Once you’re signed in, go to your profile page and enable builds for your plugin

  • And that’s it! Every time you make a commit or a PR, tests will be run against the tests-passed version

To make sure that new discourse releases don’t break your plugin, you can try setting up Travis’s CRON to run the tests every day, regardless of whether you’ve changed the plugin. My setup looks like this:

QUnit tests won't pass in discourse_dev docker image
(Vinoth Kannan) #2

Can you please tell why you allow_failure on stable branch instead of beta branch?

(David Taylor) #3

Ideally I don’t want to allow_failure on any of the 3 release branches. Unfortunately the stable branch of discourse currently fails its own tests (see all the red crosses here) and has been failing since at least 6th Feb. That’s kinda ironic since it’s supposed to be the stable branch :wink:

So once that’s fixed I’ll turn off allow_failure

(David Taylor) #4

Getting plugin CI tests working on travis is now super easy with the recent changes to the docker_test rake task. I’ve update the gist in my top post. discourse-chat-integration currently takes about 3.5 mins to load the image, setup, run rspec tests & qunit tests, so pretty quick by travis standards.

@team could someone please make the top post a wiki so I can update it? :slight_smile: Also, would you accept PRs to add travis to official plugin repos?

(Angus McLeod) #5

@david This script didn’t work for me out of the box. Is there any other config it assumes?

It fails in Travis with

The command "docker run -e "COMMIT_HASH=origin/tests-passed" -e "SKIP_CORE=1" -e SINGLE_PLUGIN=$plugin_name -v $(pwd):/var/www/discourse/plugins/$plugin_name discourse/discourse_test:release" exited with 1.

(David Taylor) #6

Hmmm… I don’t think there should be anything required beyond the Travis config above.

There’s a working config on discourse-chat-integration: Travis CI - Test and Deploy Your Code with Confidence

Do you have a link to a failing Travis job you can spare?

(Angus McLeod) #7

Yup: Travis CI - Test and Deploy Your Code with Confidence

(David Taylor) #8

Looks like it’s failing at the rubocop linting stage - roughly line 710 in the log.

If you don’t want to lint the code, and just run tests, you can add -e SKIP_LINT=1 to the docker run command.

All the environment variables available are listed at the top of docker.rake:

(Angus McLeod) #9

Ah. I had assumed they were warnings rather than errors that would stop the script. Thanks for taking a look.

(David Taylor) #10

No problem :slight_smile:

The docker script is designed to “fail quickly” - if any of the steps fail, it will exit with an error, and won’t bother with the rest of the steps.

This is good from an efficiancy point of view, but it can get quite annoying when debugging!

The length of the logs it produces is also obscene, which doesn’t help :confounded:

(Angus McLeod) #11

Sorry to keep peppering you with questions, but I’ve found that when a plugin includes a gem, like my locations plugin does, gem install [gem] will throw a permissions error when Discourse attempts to install it in the Docker container. Any thoughts on how to set the permissions right inside the container?

See: Travis CI - Test and Deploy Your Code with Confidence

Note that lib/plugin_gem.rb handles the installation of plugin gems (I think).

(David Taylor) #12

Because we’re mounting the plugin code as a volume in the docker container, it gets all of its permissions/owner/group from the host system (travis). The UID of the travis user, and the UID of the “discourse” user inside the container are not the same, and so the discourse container does not have write access to the plugin directory.

I would never recommend this in a production system, but since this is just tests, I think it is safe to do a chmod -R 777 on the plugin files before running the tests. I wouldn’t call it a fix, but it does work around the problem.

I submitted a PR to your repo with the 1 line fix, which works for travis builds on my fork :slight_smile:

I’ll add a note to the OP here in case anyone else has the same issue

(Leo McArdle) #13

@david thanks so much for the comprehensive post, it helped immensely when getting my own plugin’s specs running in Travis :heart:

Here’s an example: Travis CI - Test and Deploy Your Code with Confidence, and the scripts common to all my plugins live in:

From implementing this (and banging my head against my desk a lot) I’ve got a few improvements to propose:

Extract plugin name from plugin.rb

We have a discrepancy between the name of our plugins (e.g. mozilla-iam) and the name of its repo (e.g. discourse-mozilla-iam).

So I came up with a little oneliner to extract that name from plugin.rb rather than the directory:

(some extra escapes will have to be thrown in to put it in a .travis.yml)

Travis log folding

Like Travis does automatically with all the pre-install and pre-script gubbins, you can force Travis to fold a section of output by echoing travis_fold:start:#{id} and travis_fold:end:#{id}

So like:

echo "travis_fold:start:discourse_setup"


echo "travis_fold:end:discourse_setup"

I did however spend far too long getting this working, because I managed to misspell discourse in the end command without noticing :expressionless:

To actually wrap the relevant sections with that command I had to roll my own near-duplicate of discourse/docker.rake at master · discourse/discourse · GitHub.

I’d propose that script puts some fold commands if TRAVIS=true.

Measure code coverage and export to Coveralls

This is probably the least supported thing I got kinda working, and probably requires further discussion in its own topic.

It can be seen in practice here: mozilla/discourse-mozilla-iam | Coveralls - Test Coverage History & Statistics

The measuring of coverage itself is a bit quirky. Presumably because of the order in which files are loaded, any code executed in plugin.rb doesn’t get picked up by simplecov. I’m sure there’s some way around this, but I haven’t been able to figure it out (and I’ve just got around the problem by moving all the code I can into separate files).

To actually get coverage running, I import this file at the top of all my specs:

Which relies on those gems being installed, which I achieve like so:

It would be :ok_hand: to see support for this in core. There’s a smattering for some of it already:

But simplecov isn’t in the Gemfile.

To then get coverage exported to Coveralls, I needed to export a couple of environment variables to docker.

Happy to submit a PR incorporating some of these improvements upstream, but I’ll take guidance on which would be welcomed.

Also, if anyone has any ideas about how to check JS test coverage that would be welcomed by me! There seemed to be a few methods of getting LCOV-formatted data out of QUnit-run tests, but none seemed to slot nicely into our existing tooling.

(David Taylor) #14

Amazing! Glad my work was a useful starting point, but it sounds like you’ve made it a lot better!

I like the system you have for keeping all the travis stuff in a separate repo, rather than having to copy the .travis.yml file around. It would be nice to have an ‘official’ version of this for plugin developers to use (and for us to use in core plugins).

Cool! I had no idea you could control the folding just by echoing a string! A pull request to add this would be welcome, as long as it is behind an ENV variable.

This is interesting, I think it’s probably worth starting a new topic to discuss any core changes that would be useful for coverage reporting.

(David Taylor) #20

I’ve now set up an ‘official’ repository for testing plugins on travis. The instructions above have been updated accordingly.

I have also added the Travis ‘log folding’ to our docker.rake task, so the travis logs are much nicer to read. The 9000 line log now folds down to this


Somehow the SKIP_LINT enviroment variable is not working i guess…

even though it’s setting the variable
cause it’s still running in the building/testing process

even after terminating all the lint-errors the build is still exiting with 1 :notsureif:

The only error i’ve found so far is that prettier exiting with a 1 as well

(David Taylor) #22

Good catch, I’ve fixed it here:

If you restart your travis job it should work now :slight_smile:


Oh well yea thank you very much. Now my build is passing :slight_smile:

(David Taylor) #24

Glad to hear it. I guess you are already aware - but just in case: you have no tests, and you have now disabled linting. So the travis build is checking… nothing :wink:


Yea i am aware of that. Tests are following now, after the basic is running without linting and prettier :smiley:
Tests won’t run with blocking testers :smile: