sometimes nothin' can be a real cool hand

« Mile high code | Main | March Design Improvement EAT Workshop »

How many tests do you need?

Listen to this article Listen to this article

The number of tests you need for your system depends on how complex it is. Obvious, right? The tricky part is knowing how to measure that. There are a couple of good mathematical definitions that can help.

The Cyclomatic Complexity of a method is the number of linearly independent paths through a program's source code. This means that the Cyclomatic Complexity of a method is the minimum number of unit tests needed to test every path in a method.

Take this piece of code as an example:


public static String getFormattedName(String givenName, String middleName, String surname) {
    if (givenName != null) {
        givenName = givenName.trim();
    } else {
        givenName = "";
    }

    if (middleName != null) {
        middleName = middleName.trim();
    } else {
        middleName = "";
    }

    if (surname != null) {
        surname = surname.trim();
    } else {
        surname = "";
    }

    return givenName + " " + middleName + " " + surname;
}

Every method starts out with a default Cyclomatic Complexity of one. The complexity is then increased by one for every linearly independent path through that method. In this case, each if statement in the method represents an extra path. So the total Cyclomatic Complexity for the method is four.

In other words, if you were writing tests for this method, you should write at least four of them.

The NPath Complexity of a method is the number of acyclic execution paths through that method. This means that the NPath Complexity basically represents all of the combinations of paths possible through a method, and is an upper bound on the number of tests you need to write for a method.

Lets use the same example code again.

Every method starts out with a default NPath Complexity of one. The complexity is then increased by one for every nested conditional loop, and multiplied by two for every parallel conditional loop. In this case, the method starts out with a complexity of one. It is multiplied by two at the first if block, bringing it up to two. It is multiplied by two again at the second if block, bringing it up to four. It is multiplied by two yet again at the last if block, bringing it up to eight.

In other words, if you were writing tests for this method, you should write up to eight of them.

So the number of tests you need to write is:

Cyclomatic Complexity <= Number of Tests <= NPath Complexity

The hard part now is to try and calculate what your complexity is. Fortunately, Complexian can do that for you.

Comments

That's great for procedural code but what about object-oriented code? Cyclomatic complexity does not take polymorphic dispatch into account. I can write OO code that has a cyclomatic complexity of 1 but is branching all over the shop.

I don't see how anything changes. If I subclass a type and override a method, I need to write a test for it. I don't get to live on the one test I write for the implementation in the base class.

Can you give me an example to explain if you think that doesn't hold true?

This assumes 100% code coverage is a goal. While I can see that this assumption could be good I think it is worth stating as an assumption.

Sometimes from a cost-benefit analysis, 100% coverage is not appropriate, even to test-driven zealots like myself.

Actually, I think coverage tools use a different measure. I suspect you could get 100% coverage using some tools and write less tests. I also suspect you get write the number of tests I'm suggesting and not get 100% coverage (though that would be harder).

I'm probably being a bit picky. Your intent is correct. I guess what I'm saying is "To cover every path through your code, you need to write this many tests".

Post a comment