Multiple Asserts in a Single Unit Test method

The title of this post is a provocation to many people who have read and love Roy Osherove’s brilliant book, The Art of Unit Testing. In this book Roy clearly states that one of the pillars of good tests is to avoid multiple asserts in a unit test.

The arguments against multiple asserts are multiple, a main one being that if one assert fails the rest will not be executed, which means that the state of the code under unit test is really unknown. Another argument is that if you find a need for having multiple asserts, it is probably because you are testing multiple things in a single unit test method. This will break the principle of single responsibility and maintainability will suffer.

I am a great believer in having maintainable and readable unit tests and I have always tried to follow the single assert advise myself. I am also a great believer in the principle of single responsibility, although I am often forced to be pragmatic when working on legacy code. When I want to test several outcomes from a single object I can choose to implement Equals, or maybe ToString in order to do direct comparisons of whole objects. Sometimes I will try to make a utility method or class that will allow me to compare several values in a way that will fit a single assert. While some people do not like adding to the code base for unit testing purposes only, most people object to having too many utilities creeping up in the unit test projects.

Recently I had discussions on unit testing with my colleagues and the reasoning behind single asserts came up – and also some arguments against it.

Let’s have a look at one of Roy’s examples,

[TestMethod]
public void CheckVariousSumResults()
{
    Assert.AreEqual(3, this.Sum(1001, 1, 2));
    Assert.AreEqual(3, this.Sum(1, 1001, 2));
    Assert.AreEqual(3, this.Sum(1, 2, 1001));
}

The problem here is that if one assertion fails, the rest will not be run and we do not know if they would fail if run.

There are a number of solutions to this.

The first solution: Create a separate test for each assert

This is easy and it only takes a few seconds to write those unit tests,

[TestMethod]
public void Sum_1001AsFirstParam_Returns3()
{
    Assert.AreEqual(3, this.Sum(1001, 1, 2));
}
[TestMethod]
public void Sum_1001AsMiddleParam_Returns3()
{
    Assert.AreEqual(3, this.Sum(1, 1001, 2));
}
[TestMethod]
public void Sum_1001AsThirdParam_Returns3()
{
    Assert.AreEqual(3, this.Sum(1, 2, 1001));
}

What is the problem with this solution?

Well, although the example may be slightly contrived it is easy to imagine that the three cases are somewhat correlated. By putting all three asserts in a single method we have signaled that these must be considered as a whole in order to be understood, while if we create separate unit test we have lost this information. And imagine that there were more than three cases, say 42? If a fundamental bug in the Sum method creeps in so that all 42 unit tests fail, would you prefer to have 42 unit tests fail or would you prefer to have a single unit test fail?

Another problem is maintainability. It is correct that it only takes a few seconds to write these three unit tests, but someone needs to maintain them in all future and the task can become daunting due to the sheer number of unit tests.

Both problems can to a certain extend be overcome with proper naming and with true single responsibility of units under test as well as each unit test method, but that is not always the reality – especially when you try to put legacy code under unit test.

The Second Solution: Use Parameterized Tests

In many cases I would prefer to use parameterized tests. However, currently my unit testing environment is Visual Studio 2010 and it does not support such a feature!

The Third Solution: Use try-catch

Since an assertion failure means that an exception is thrown, at least in the unit test frameworks I have used so far, we can simply catch such exceptions, do some intelligent processing, and then allow the next assertion to fail or succeed. That solves our problem with having multiple asserts.

Even though Roy is my unit testing hero, I think he is a bit too hasty to simply abandon the try-catch solution with a statement like "Some people think it’s a good idea to use a try-catch block [...] I think using parameterized tests is a far better way of achieving the same thing."

A Simple Solution Using try-catch

Since I cannot write parameterized unit tests with my unit testing environment, I had to come up with an alternative solution. My solution is to introduce a new MultiAssert class which will accept delegates to multiple assert statements but only fail at most once. This new class seems to be a logical addition to the existing family of assert classes along with e.g. CollectionAssert and StringAssert.

Here is the above example in a single unit test with a single assertion,

[TestMethod]
public void CheckVariousSumResults()
{
    MultiAssert.Aggregate(
        () => Assert.AreEqual(3, this.Sum(1001, 1, 2)), 
        () => Assert.AreEqual(3, this.Sum(1, 1001, 2)), 
        () => Assert.AreEqual(3, this.Sum(1, 2, 1001)));
}

MultiAssert.Aggregate can even be used in situations that do not fit parameterized unit tests easily.

Here is the implementation of MultiAssert.

public static class MultiAssert
{
    public static void Aggregate(params Action[] actions)
    {
        var exceptions = new List<AssertFailedException>();

        foreach (var action in actions)
        {
            try
            {
                action();
            }
            catch (AssertFailedException ex)
            {
                exceptions.Add(ex);
            }
        }

        var assertionTexts = 
            exceptions.Select(assertFailedException => assertFailedException.Message);
        if (0 != assertionTexts.Count())
        {
            throw new
                AssertFailedException(
                assertionTexts.Aggregate(
                    (aggregatedMessage, next) => aggregatedMessage + Environment.NewLine + next));
        }
    }
}

Using or Abusing Multiple Asserts

MultiAssert can be abused, it is not meant as a universal excuse for cramming a lot of assertions into any unit test method. Remember that maintainability and readability of unit tests must still be a top priority and you should only use MultiAssert when this can be achieved.

One situation in which I recommend the use of MultiAssert is when it makes sense to assert both pre- and post-conditions in a unit test method. In this context, a post-condition is simply a (single) assert that states something about the state of the world after the Act part of the unit test method. However, if you assert that something has the value 42, how do you know that this was not already true right after the Arrange part of the unit test? After all, the Assert part of your unit test must assert what was supposed to happen as a consequence of the Act part of the unit test method.

So one nice usage of MultiAssert is to assert both pre- and post-conditions in unit tests.

[TestMethod]
public void Foo()
{
    // Arrange
    var underTest = …;
    bool preCondition = underTest.TheFoo() != 42;

    // Act
    underTest.Foo();
    int actual = underTest.TheFoo();

    // Assert
    MultiAssert.Aggregate(
        () => Assert.IsTrue(preCondition),
        () => Assert.AreEqual(42, actual));
}