The Skeptical Methodologist

Software, Rants and Management

Combine Testing Strategies for Increased Quality with Less Work

Testing and correctness techniques build off of each other, and rarely replace each other.  I was reminded of this fact when an argument over the merits of unit testing versus Design by Contract.  The argument was that invariants and preconditions that were being checked in code should be moved out into unit tests and exercised explicitly.

There are multiple problems with this.  The best way to describe the overarching theme of wrongness, here, though is the “black swan” problem.  That is, you can never prove that there are no black swans without individually looking at every single swan out there (let’s just pretend its impossible.)  You can, however, prove that there is at least one white swan just by finding one.  Unit tests, especially TDD style unit tests, are checking for white swans.  They put together certain scenarios where the output is a well known value, and then ensure that the program, as implemented, outputs that well known value.  Unit tests should not be looking for black swans because, due to their nature, it’s incredibly hard to write enough tests to cover ALL possibilities of black swans.  Invariants, for instance, are black swan sorts of problems.

If I claim that “after I do x, y and z, my state should move from U to V”, then I’m describing a white swan.  I only have to do it once to know that, in this situation, it was true.  But if I say something like “I should never be in state U”, then I’m talking about a black swan, and that’s something that cannot be tested very thoroughly.  Sure, maybe I don’t go into state U given the stimulus of my test, but what’s to say I never go into state U?  I need a different strategy.

One of these strategies is type checking, and it does a very good job of rooting out black swans, but only for certain well formed ideas.  I can prove that I never get an Float when I expect a String, but many things can’t be proven in this way.

Another strategy is invariant enforcement in Design-By-Contract style assertions.  These will not prove that there are no black swans, but instead are sort of a ‘brute force’ approach.  I said earlier that it’s nigh impossible to find all swans and make sure they aren’t black – but if I already have a hundred or so tests, or perhaps many hours of real life use of my application, certainly I should check any swans I find THERE are not black.  These assertions will give me some evidence, at least, that my lakes are relatively black swan free.

Long story short, Design-By-Contract is a good way to do validity testing (“Did we build the thing right?”) and TDD style unit test are a good way to do verification testing (“Did we build the right thing?”).  But more importantly, put together, they are more powerful than either one alone.  TDD looks for white swans, while DBC asserts the non-existence of black ones.  TDD already has done a lot of work to find certain white swans, why not make sure, in the code, you guarantee they aren’t black swans while you’re out?  And why not, to reduce code duplication and increase self-documentation of the code, you put the assertions that you haven’t found a black swan where the black swan might appear, rather than in a test somewhere else that checks for its existence?

In other words, your test coverage is driven by your unit tests.  The more unit tests you have, the higher your test coverage (hopefully).  But coverage does not only have to ensure that you found all the white swans you anticipated, but that you found no black ones as well.  Another test enhances the quantity of your coverage, but strategies like DBC and other assertions enhance the quality.  The two strategies remain separate, though, since they have different concerns.


April 20, 2009 - Posted by | Uncategorized

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: