Solving the “Problems” with TDD

First, read Dalke Scientific’s criticisms of TDD, or Test-Driven-Development.

The important thing to note is that when we talk about TDD, we’re talking about writing a test to drive the design of an API, you ‘test first’ before the implementation is written.  This is often juxtaposed to testing after an implementation is known.

This is an important thing to note, as I think taking DS’s criticism’s in this light makes sense of many of the problems the author finds with TDD approaches.  Namely, as is often said, TDD should not supplant many other testing strategies for code.  DS points out, rightly so, that some authors like Uncle Bob and Kent Beck have oversold TDD as yet another silver bullet.  Not all steps to the development process naturally flow from TDD.  In fact, as some have pointed out, TDD has little rigorous advice towards incorporating domain knowledge, or exploring a domain in which you have very little experience.  You can’t TDD your way through a Sudoku solver, or a ray tracer.  Instead, you need to understand the algorithms behind those solutions.

TDD is specifically about driving clean, reusable interfaces from well understood requirements.  Hah!  Since when are requirements ever well understood?  But that’s ok, since TDD is frequently packaged with a good dose of refactoring, we should expect any accomplished TDD’er to also respond well to changing requirements.  So, let’s rephrase that – TDD is about driving clean, reusable interfaces from requirements.  That’s all – it is no guarantee of code quality.  Furthermore, as DS points out, it is no guarantee of code coverage either!  Simplistic TDD would state that no line would be constructed without a test requiring its behavior.  That is true.  But recall, as I just stated, TDD is packaged with refactoring, which itself (as DS rightly notes) makes no guarantee on test coverage.  On an opposite note, one place where I disagree with DS is where he makes a big deal out of the performance requirements of finding Prime numbers.  TDD solutions are (or should be) entirely derived from requirements since they more or less are code implementation test forms of the requirements.  If Uncle Bob’s self-imposed requirements on his prime finder don’t specify processing speed, then his approach is legitimate.  What DS should have argued is that in the case of algorithmic code like a prime finder, generally speaking performance matters and would have been specified formally in one way or another (which implies that performance should be tested under a TDD scheme and would have ruled Uncle Bob’s solution out.)  I do sympathize with DS’s aesthetic revulsion to Uncle Bob reinventing an already understood algorithm in a slower form to prove that TDD works, but that’s a side issue.

So, I’d like to reset the debate around this claim:  TDD is a good “enterprise” technique to drive clean, reusable interfaces with high, but not perfect, code coverage.  I use the dreaded term “enterprise” here to mean “corporate” style requirements that are somewhat well understood versus blue-sky or academic research requirements, and they also tend to be brute force heavy and algorithmically light.  Oddly enough the ‘plumbing’ code this description provides makes up a huge percentage of what we, as software developers, are expected to write.  So TDD certainly has its place.

Where is that place?  Let’s go a little into testing theory real quick and it will jump out at us.  Traditionally, there are ‘verification’ tests and ‘validation’ tests.  Verification asks “did we build the thing right?”, while validation asks “did we build the right thing?”.  Verification asks questions about quality, about coverage, about safety and reliability.  It makes sure that, whatever we built, we did it ‘well’, that it has ‘quality’ or ‘craftsmanship’.  Validation, on the other hand, asks whether the thing we built was the thing the customer asked for in the first place.

There’s another dimension to testing, commonly referred to as white-box vs black-box.  Black box tests treat the implementation as an unknown and strictly tests the interface.  White box techniques look at the implementation specifically and try to break it.  They are both beneficial as looking at software as a black box generally allows one to test it a lot more aggressively since they have no ‘clues’ via the implementation to try to break it, so they just throw everything they have at boundaries and faults.  White box is beneficial for exactly the opposite reason, the implementation gives the tester hints on what might break via reverse reasoning (assume it broke and then work backwards to see what inputs would break it.)  Black box testing is also easier to automate ‘tests’ for, like fuzz or smoke tests, while white box styles allow better ‘static analysis’ techniques.

You ‘need’ to cover all 4 quadrants of testing for a good strategy.  DS points out the ‘flaws’ in TDD, but in the context of testing theory, these are not flaws.  The characteristics of TDD put it squarely in the “black box validation” camp, along with techniques like requirements traceability and use case analysis.  I’d say in that company, it does quite well.   The “traditional” unit tests DS refers to fall more in the “white box verification” camp, i.e., you know the implementation and you’re specifically checking it for quality.  This quadrant needs to be supported to, and there are a myriad of techniques to do so (some of my favorite include Joel’s ‘completely separate testing team’ approach as well as Design-By-Contract.)

So, to sum up, the “problems” with TDD aren’t problems at all: they’re endemic to the quadrant of testing TDD is associated with.  TDD has been oversold as a magic bullet by its promoters, that is for certain, but it is not synonymous with “good testing” as the original author states.  It has a very specific place in a “good testing” strategy, namely, taking requirements and directly stating them as automated validation tests of any potential solution’s interface.