Thursday, April 21, 2011

How much are you willing to pay to speed up your Rails tests?

The trending topic on twitter lately is about how fast can you make your tests run. Unfortunately what never gets mentioned in those twitter conversations is balance; the balance between speed and good specification of behavior.

Attila Domokos has been working on speeding up his tests and documented his efforts in his blog. While the effort is good, I have concerns about this type of approach and it's associated cost.

Unfortunately this approach relies heavily on mocking and stubbing, and not just to fake out one method, but several methods and objects. Mocking methods ties your tests tightly to the internal implementation of the object you are mocking at which point you are no longer testing behavior, but you are testing implementation. This is a serious concern and leads to brittle tests. This should be avoided.

My other concern is that this approach relies heavily on good integration level testing. Integration testing is cumbersome and I've yet to see a suite with more than a handful of tests that run in under a minute. Often times these test suites take 30-60 minutes to run for an application of even moderate size. So if our goal to keep our test suite running in under 10 seconds, all we've done is pushed our longer running tests into an integration suite that has a much larger feedback loop in exchange for brittle tests that save us a few hundredths of a second. It's like using a credit card. The immediate purchase was satisfying, however the debt collection was costly.

Now there are tools like spork and auotest that can help get shorter feedback loops and i suggest your team explore those options first before sacrificing the quality of your tests.

What I ask of the community is that when you're working with your test suite and trying to speed it up, please acknowledge any concessions you may be making in the name of speed and make a smart decision about how you proceed.