(Superseded) Set up plugin continuous integration tests on Travis CI

david · March 21, 2017, 11:52am

This is out-of-date. Now, we recommend using GitHub Actions for this task. See guide.

Following on from Beginner’s Guide to Creating Discourse Plugins Part 6: Acceptance Tests

To set up continuous integration for your plugin, you can use our discourse-plugin-ci repository. This will run the Discourse core linters (prettify, eslint & rubocop), your plugin’s Rspec tests, and your plugin’s Qunit tests.

For an example, check out discourse-chat-integration

Setting it up

Copy & paste .travis.yml from here to the root directory of your plugin’s repository.
Copy & paste .rubocop.yml from here to the root directory of your plugin’s repository.
Head over to http://travis-ci.org, and sign up with your GitHub account
Once you’re signed in, go to your profile page and enable builds for your plugin
And that’s it! Every time you make a commit or a PR, tests will be run against the tests-passed version

Cron Job

To make sure that new discourse releases don’t break your plugin, you can try setting up Travis’s CRON to run the tests every day, regardless of whether you’ve changed the plugin. My setup looks like this:

Additional Docker arguments

Do you need to supply additional arguments to the Docker container? You can use the DOCKER_OPTIONS environment variable for that.

For example, if your plugin depends on another plugin, clone it and export the environment variable in .travis.yml.

before_script:
  - git clone --depth=1 https://github.com/discourse/discourse-plugin-ci
  - git clone --depth=1 <git_clone_url> $HOME/plugins/<additional_plugin_name>

script:
  - export DOCKER_OPTIONS="-v ${HOME}/plugins/<additional_plugin_name>:/var/www/discourse/plugins/<additional_plugin_name>"
  - discourse-plugin-ci/script.sh

vinothkannans · March 21, 2017, 1:02pm

Can you please tell why you allow_failure on stable branch instead of beta branch?

david · March 21, 2017, 1:14pm

Ideally I don’t want to allow_failure on any of the 3 release branches. Unfortunately the stable branch of discourse currently fails its own tests (see all the red crosses here) and has been failing since at least 6th Feb. That’s kinda ironic since it’s supposed to be the stable branch

So once that’s fixed I’ll turn off allow_failure

david · July 21, 2017, 11:41am

Getting plugin CI tests working on travis is now super easy with the recent changes to the docker_test rake task. I’ve update the gist in my top post. discourse-chat-integration currently takes about 3.5 mins to load the image, setup, run rspec tests & qunit tests, so pretty quick by travis standards.

https://gist.github.com/davidtaylorhq/0c73c93af68724ffbd47f18eff7fbb12

@team could someone please make the top post a wiki so I can update it? Also, would you accept PRs to add travis to official plugin repos?

angus · September 5, 2017, 6:54am

@david This script didn’t work for me out of the box. Is there any other config it assumes?

It fails in Travis with

The command "docker run -e "COMMIT_HASH=origin/tests-passed" -e "SKIP_CORE=1" -e SINGLE_PLUGIN=$plugin_name -v $(pwd):/var/www/discourse/plugins/$plugin_name discourse/discourse_test:release" exited with 1.

david · September 5, 2017, 8:52am

Hmmm… I don’t think there should be anything required beyond the Travis config above.

There’s a working config on discourse-chat-integration: Travis CI - Test and Deploy Your Code with Confidence

Do you have a link to a failing Travis job you can spare?

angus · September 5, 2017, 8:55am

Yup: Travis CI - Test and Deploy Your Code with Confidence

david · September 5, 2017, 9:00am

Looks like it’s failing at the rubocop linting stage - roughly line 710 in the log.

If you don’t want to lint the code, and just run tests, you can add -e SKIP_LINT=1 to the docker run command.

All the environment variables available are listed at the top of docker.rake:

https://github.com/discourse/discourse/blob/master/lib/tasks/docker.rake

angus · September 5, 2017, 9:07am

Ah. I had assumed they were warnings rather than errors that would stop the script. Thanks for taking a look.

david · September 5, 2017, 9:10am

No problem

The docker script is designed to “fail quickly” - if any of the steps fail, it will exit with an error, and won’t bother with the rest of the steps.

This is good from an efficiancy point of view, but it can get quite annoying when debugging!

The length of the logs it produces is also obscene, which doesn’t help

angus · September 20, 2017, 3:36am

Sorry to keep peppering you with questions, but I’ve found that when a plugin includes a gem, like my locations plugin does, gem install [gem] will throw a permissions error when Discourse attempts to install it in the Docker container. Any thoughts on how to set the permissions right inside the container?

See: Travis CI - Test and Deploy Your Code with Confidence

Note that lib/plugin_gem.rb handles the installation of plugin gems (I think).

david · September 20, 2017, 4:02pm

Because we’re mounting the plugin code as a volume in the docker container, it gets all of its permissions/owner/group from the host system (travis). The UID of the travis user, and the UID of the “discourse” user inside the container are not the same, and so the discourse container does not have write access to the plugin directory.

I would never recommend this in a production system, but since this is just tests, I think it is safe to do a chmod -R 777 on the plugin files before running the tests. I wouldn’t call it a fix, but it does work around the problem.

I submitted a PR to your repo with the 1 line fix, which works for travis builds on my fork

I’ll add a note to the OP here in case anyone else has the same issue

LeoMcA · August 17, 2018, 2:32pm

@david thanks so much for the comprehensive post, it helped immensely when getting my own plugin’s specs running in Travis

Here’s an example: Travis CI, and the scripts common to all my plugins live in:

From implementing this (and banging my head against my desk a lot) I’ve got a few improvements to propose:

Extract plugin name from `plugin.rb`

We have a discrepancy between the name of our plugins (e.g. mozilla-iam) and the name of its repo (e.g. discourse-mozilla-iam).

So I came up with a little oneliner to extract that name from plugin.rb rather than the directory:

github.com

mozilla/discourse-mozilla-travis/blob/master/before_script.sh#L2


      
          #!/bin/bash
          export PLUGIN_NAME=$(sed -n -e "s/# name: \([^ ]*\)\s*/\1/p" plugin.rb)
          echo "PLUGIN_NAME=$PLUGIN_NAME"
          chmod -R 777 . # This is necessary if your plugin installs gems

(some extra escapes will have to be thrown in to put it in a .travis.yml)

Travis log folding

Like Travis does automatically with all the pre-install and pre-script gubbins, you can force Travis to fold a section of output by echoing travis_fold:start:#{id} and travis_fold:end:#{id}

So like:

echo "travis_fold:start:discourse_setup"

and

echo "travis_fold:end:discourse_setup"

I did however spend far too long getting this working, because I managed to misspell discourse in the end command without noticing

To actually wrap the relevant sections with that command I had to roll my own near-duplicate of discourse/lib/tasks/docker.rake at main · discourse/discourse · GitHub.

I’d propose that script puts some fold commands if TRAVIS=true.

Measure code coverage and export to Coveralls

This is probably the least supported thing I got kinda working, and probably requires further discussion in its own topic.

It can be seen in practice here: mozilla/discourse-mozilla-iam | Coveralls - Test Coverage History & Statistics

The measuring of coverage itself is a bit quirky. Presumably because of the order in which files are loaded, any code executed in plugin.rb doesn’t get picked up by simplecov. I’m sure there’s some way around this, but I haven’t been able to figure it out (and I’ve just got around the problem by moving all the code I can into separate files).

To actually get coverage running, I import this file at the top of all my specs:

github.com

mozilla/discourse-mozilla-travis/blob/master/plugin_helper.rb

if ENV["COVERALLS"] || ENV["SIMPLECOV"]
  require "simplecov"

  if ENV["COVERALLS"]
    require "coveralls"
    SimpleCov.formatter = Coveralls::SimpleCov::Formatter
  end

  SimpleCov.start do
    root File.expand_path("../..", __FILE__)
    add_filter "spec/"
    add_filter "db/migrate"
    add_filter "gems/"
  end
end

require "rails_helper"

Which relies on those gems being installed, which I achieve like so:

github.com

mozilla/discourse-mozilla-travis/blob/master/entrypoint.sh#L8-L9


      
          git fetch origin tests-passed -q || exit 1
          git reset --hard origin/tests-passed -q || exit 1

It would be to see support for this in core. There’s a smattering for some of it already:

github.com

discourse/discourse/blob/main/spec/rails_helper.rb#L1-L4


      
          # frozen_string_literal: true
          
          if ENV["COVERAGE"]
            require "simplecov"

But simplecov isn’t in the Gemfile.

To then get coverage exported to Coveralls, I needed to export a couple of environment variables to docker.

Happy to submit a PR incorporating some of these improvements upstream, but I’ll take guidance on which would be welcomed.

Also, if anyone has any ideas about how to check JS test coverage that would be welcomed by me! There seemed to be a few methods of getting LCOV-formatted data out of QUnit-run tests, but none seemed to slot nicely into our existing tooling.

david · August 17, 2018, 3:00pm

Amazing! Glad my work was a useful starting point, but it sounds like you’ve made it a lot better!

I like the system you have for keeping all the travis stuff in a separate repo, rather than having to copy the .travis.yml file around. It would be nice to have an ‘official’ version of this for plugin developers to use (and for us to use in core plugins).

Cool! I had no idea you could control the folding just by echoing a string! A pull request to add this would be welcome, as long as it is behind an ENV variable.

This is interesting, I think it’s probably worth starting a new topic to discuss any core changes that would be useful for coverage reporting.

david · September 13, 2018, 5:23pm

I’ve now set up an ‘official’ repository for testing plugins on travis. The instructions above have been updated accordingly.

I have also added the Travis ‘log folding’ to our docker.rake task, so the travis logs are much nicer to read. The 9000 line log now folds down to this

Machigatta · November 21, 2018, 2:05pm

Somehow the SKIP_LINT enviroment variable is not working i guess…

even though it’s setting the variable
cause it’s still running in the building/testing process

even after terminating all the lint-errors the build is still exiting with 1

The only error i’ve found so far is that prettier exiting with a 1 as well
https://travis-ci.org/kokoro-ko/discourse-humble-box/builds/457954010

david · November 21, 2018, 2:09pm

Good catch, I’ve fixed it here:

https://github.com/discourse/discourse-plugin-ci/commit/006082b794bb560adaa7f9a5e0925ab5aa79e3e1

If you restart your travis job it should work now

Machigatta · November 21, 2018, 2:17pm

Oh well yea thank you very much. Now my build is passing

david · November 21, 2018, 2:20pm

Glad to hear it. I guess you are already aware - but just in case: you have no tests, and you have now disabled linting. So the travis build is checking… nothing

Machigatta · November 21, 2018, 2:21pm

Yea i am aware of that. Tests are following now, after the basic travis.ci is running without linting and prettier
Tests won’t run with blocking testers

Topic		Replies	Views
Developing Discourse Plugins - Part 6 - Add acceptance tests Developer Guides testing , plugin-guides , tutorial	32	16447	June 2, 2022
Install Discourse for development using Docker Developer Guides docker , dev-install , how-to	204	71376	October 22, 2024
Install Discourse on an isolated CentOS 7 server Sysadmins how-to , install	23	11047	July 5, 2022
Developing Discourse Plugins - Part 1 - Create a basic plugin Developer Guides plugin-guides , tutorial	33	77011	October 24, 2024
Rails plugin generator Announcements	50	12509	August 20, 2021

(Superseded) Set up plugin continuous integration tests on Travis CI

Setting it up

Cron Job

Additional Docker arguments

Extract plugin name from plugin.rb

Travis log folding

Measure code coverage and export to Coveralls

Related Topics

Extract plugin name from `plugin.rb`