How can we speed up our test suite?

(Sam Saffron) #1

At the moment, running our test suite takes upwards of 8 minutes on travis for only 1861 examples.

More an more tests are being added daily and the test suite is getting slower and slower.

What strategies/refactoring can we make to improve the performance of our test suite?

As always, pull requests welcome.

(Alexander) #2

There’s a seed happening each time, right? Getting rid of that might shave off a minute.

(Robin Ward) #3

Damn, it’s just 2:18 on my machine. That’s on travis right?

(Alexander) #4

Wow. On my MBP, 2.66 GHz i7, 8 GB 1067 MHz, spinning drive:

Finished in 5 minutes 26.28 seconds

1861 examples, 0 failures

(Sam Saffron) #5

Yeah travis, our build box is about 6 minutes 1 second, my VM is 2 minutes 25 seconds. But I am cheating. (should update to the latest funny falcon)

(Robin Ward) #6

I’m on a Hackintosh sandy bridge i7 + fast SSD, so that probably has something to do with it :slight_smile:

(Mike Moore) #7

Unit tests have the biggest RIO, so I would focus more on them. Let the pain of cumbersome integration test setup and slower text execution drive changes in your design. Full-stack acceptance tests should be used sparingly, and any failure in acceptance tests should also be accompanied by a failing unit test that shows you exactly where the problem is.

Integration Tests Are a Scam

(Sam Saffron) #8

A typical issue performance wise I see is:

In this test we are create a real topic 2 times and views 3 times, reloading a user from the db 2 times just to test that topics_entered is correct.

Our specs are often designed to test isolated behaviours but do them on real object that live in the DB and create tons of side effects (thus testing them)

The setup is usually “not shared”, a trivial improvement for this particular test, is to do the setup once and simply run 2 assertions in a chain.

I agree that one huge issue we have is that a lot of our tests are integration as opposed to unit ones.

(Jimmy Bogard) #9

I always hated the absolutist approach, because different kinds of tests provide different kinds of value. A previous “big project” with about 250-300 controllers and 750 or so actions resulted in a total of 6000 or so total tests.

Ratio-wise, we had:

  • 4000 unit tests
  • 1000 integration tests (i.e., tests that hit the database)
  • 500 subcutaneous tests
  • 500 functional UI tests

Timing-wise, the CI build ran for 5 minutes but would break from unit tests failure in about 30 seconds. The integration/subcutaneous tests took the remaining time.

The functional UI tests took about 45 minutes to run (on Selenium Grid), as a chained build in Jenkins. Those weren’t even red/green failures, we merely reported a percentage. Once we figured out how to write functional UI tests that weren’t brittle, those stayed at 99-100% pass rates.

Instead of thinking about terms of “XYZ kinds of tests suck”, think about what are healthy ratios. 100% unit tests won’t prevent bugs, so you have to have a balanced approach. Favor quicker tests, but don’t ignore the others - they have value too.

(Mike Moore) #10

This is the thing I see most often with folks who learned to test with RSpec. Not a dig on RSpec, but these types of tests are very common in rails codebases. There are no easy solutions without changing the approach to how these domain models are designed.

Hit me up privately and I’ll share my thoughts.

(Daniel Watkins) #11

I think we’d all be interested to hear your thoughts…

(Jeff Atwood) #12

Not me! I don’t want to hear those dirty, filthy, private RSpec testing thoughts! Keep those to yourself, man! :hear_no_evil:

(Dan Neumann) #13

Rails offers no conventions for writing tests (beyond redefining the meaning of words like “integration”, “functional”, and “unit”), and it certainly offers no conventions for writing code and tests that aren’t coupled to Rails.

I find it hard to find literature that offers advice in this area. I’ve stumbled upon things like Corey Haines’ Fast Rails Tests talk and Gary Bernhardt screencasts, but it feels like hidden, coveted wisdom that nobody is really talking about.

In other words, I’d also like to hear @blowmage share some wisdom.

(Alexander) #14

I just watched a Travis build start, and it took 3:45ish until it even got to rspec && jasmine.

(Sam Saffron) #15

A very cheap fix, based on some of my work here:

Our test suite runs on a debian VM, I played with various knobs to find the fastest way to run our tests, our baseline was 4:57 mins to run all our tests on 1.9.3 - p392

RAILS_ENV=test RUBY_GC_MALLOC_LIMIT=50000000 LD_PRELOAD=/usr/lib/ bundle exec rspec spec

Takes 3:10 mins. This is a huge gain over our vanilla run of our test suite.

Some stuff my testing revealed:

  • Debian sucks as a test machine, its just too old/stable. -O3 build was actually slower than the pre-compiled RVM binary. In my ubuntu testing it was always faster. tcmalloc is older, gcc is older and so on. Its just the wrong machine to be running your tests on.

  • Raising RUBY_GC_MALLOC_LIMIT from 50 megs to 75 megs slows the tests down by a few seconds, similarly dropping it to 25 slows it again. (Keep in mind default is 8)

I totally agree we should clean up our tests and speed up the suite. In fact, since writing we are already 30 or so seconds slower on my dev machine. That said, the super cheap wins by turning on a few turbo switches sure feels good.

Ruby 2.0 would be yet faster, maybe 5-10%, but we would need to switch our entire hosting infrastructure to 2.0, otherwise running tests on such a different environment to prd is just totally risky.

(Sam Saffron) #16

Update, after adding the following to travis.


(Sam Saffron) #17

One thing that I noticed that kind of annoys me is the almost religious following of the rule … one it tests one thing


describe 'user' do 
  let :user do 

  it "is has a username" do 
      user.username.should_not be_nil

  it "is has a username_lower" do 
      user.username_lower.should_not be_nil

  it "is has a name" do be_nil


describe 'user' do 
  let :user do 

  it "is has the correct fields set" do 
      user.username.should_not be_nil
      user.username.should_not be_nil be_nil

I think better specs does a better job than me explaining why we should be careful following this blindly:

The ‘one expectation’ tip is more broadly expressed as ‘each test should make only one assertion’. This helps you on finding possible errors, going directly to the failing test, and to make your code readable.

In isolated unit specs, you want each example to specify one (and only one) behavior. Multiple expectations in the same example are a signal that you may be specifying multiple behaviors.

Anyway, in tests that are not isolated (e.g. ones that integrate with a DB, an external webservice, or end-to-end-tests), you take a massive performance hit to do the same setup over and over again, just to set a different expectation in each test. In these sorts of slower tests, I think it’s fine to specify more than one isolated behavior.

The single test rule is fine, for stuff that does not touch the db. Once we do we should try to maximise the amount of testing for each expensive db setup.

Thing is, the majority of our tests involve some db work… so a lot of refactoring needed.

@eviltrout thoughts?

(Robin Ward) #18

The harsh sands of time have worn me down on the single test dogma. It does make finding failures much easier, but I agree sometimes the tear up / tear down are not worth it.

I should point out though that your example could be replaced with and it would be entirely memory. I think a good start would be to encourage all tests to use build if possible and avoid hitting the db altogether.

For those specs that have to hit the db to work properly, grouping is fine by me, although I think we have to use some developer judgment for readability. If you write a 20 line spec testing various things, that might be confusing to maintain. People should comment / break up specs where appropriate.

(Sam Saffron) #20

I love zeus, it is ultra awesome. I use it.

However for the test running I stick with bundle exec guard … guard has a bunch of advantages

  • I uses spork anyway, so it has a reasonably fast forker
  • It watches my files and launches tests as I change stuff
  • I have export RSPEC_FAIL_FAST=1 in my .rvm rc, this sets config.fail_fast = ENV['RSPEC_FAIL_FAST'] == "1" - it means that any single failure I get halts the test suite
  • We use focus_on_failed in our guard file, that allows you to work on a single failed test automatically. (I still need some smarter patches there, but it works ok for now)

That said I would be happy to get zeus test working as long as it does not break anything existing.

(Sander Datema) #21

For some reason Travis got stuck in testing this pull request (just a YAML file edited) Dutch translation updated by sanderdatema · Pull Request #723 · discourse/discourse · GitHub. Here’s Travis’s queue: Travis CI - Test and Deploy Your Code with Confidence

So it’s not that the tests failed, no, they’re still running. For almost an hour already.

@sam or @codinghorror, any way to force finish it? Would love to have those translations get through.