Thursday, April 21, 2011

How much are you willing to pay to speed up your Rails tests?

The trending topic on twitter lately is about how fast can you make your tests run. Unfortunately what never gets mentioned in those twitter conversations is balance; the balance between speed and good specification of behavior.

Attila Domokos has been working on speeding up his tests and documented his efforts in his blog. While the effort is good, I have concerns about this type of approach and it's associated cost.

Unfortunately this approach relies heavily on mocking and stubbing, and not just to fake out one method, but several methods and objects. Mocking methods ties your tests tightly to the internal implementation of the object you are mocking at which point you are no longer testing behavior, but you are testing implementation. This is a serious concern and leads to brittle tests. This should be avoided.

My other concern is that this approach relies heavily on good integration level testing. Integration testing is cumbersome and I've yet to see a suite with more than a handful of tests that run in under a minute. Often times these test suites take 30-60 minutes to run for an application of even moderate size. So if our goal to keep our test suite running in under 10 seconds, all we've done is pushed our longer running tests into an integration suite that has a much larger feedback loop in exchange for brittle tests that save us a few hundredths of a second. It's like using a credit card. The immediate purchase was satisfying, however the debt collection was costly.

Now there are tools like spork and auotest that can help get shorter feedback loops and i suggest your team explore those options first before sacrificing the quality of your tests.

What I ask of the community is that when you're working with your test suite and trying to speed it up, please acknowledge any concessions you may be making in the name of speed and make a smart decision about how you proceed.

1 comment:

  1. Another option is to close some shit down, or beef the hardware in the machine. Honestly, the highest contributing factor I have found in test times is how much stress the system is under to begin with.

    While our test times have been stabilized between 6 and 7 seconds on my machine lately, they skyrocketed to 10 to 12 seconds. What was different? I was running a huge data import taking data from MySQL into Mongo. The dataset is already in memory and Ruby is just pillaging the CPU for everything it has to iterate of the result set and save them into Mongo. We have also seen quite different test suite times between even slightly different hardware configurations between our development machines.

    So to add to the great info in this post, don't overlook the obvious and don't forget that something as CPU intensive as a test suite is going to be directly affected by the processing resources available for use on the machine.