Introduction
In many unit test frameworks, you have the opportunity to run some code just before a test method runs, and just after a test method finished. In NUnit, you mark such methods with [SetUp] and [TearDown] attributes; in VSTS, you use [TestInitialize] and [TestCleanup].
Writing this code (using the NUnit attributes):
[TestFixture]
public class MyTests
{
private Foo theFoo = new Foo();
[SetUp]
public void CreateFoo()
{
theFoo = new Foo();
}
[TearDown]
{
theFoo.Dispose();
}
[Test]
public void FooIsNotNull()
{
Assert.IsNotNull(theFoo);
}
}
Effectively, these methods work to set up and clean up the resources used in the test. What's not to like, right?
My Issues
In my mind, there are a few problems with this system.
1. Non-Obvious Code
It's very easy for the setup and cleanup code to get separated from the actual test being written. Just looking at the FooIsNotNull test, you don't know what the setup method has done to arrive at the value. In fact, even knowing that there is a value already made for you can sometimes be difficult.
2. Difficult Debugging
If the code in SetUp or TearDown throws, it can be difficult to determine what the problem is. If the code is in the test method, it's easier to track down the exact issue.
3. Inflexibility
By using a fixed SetUp and TearDown, there is a presumption that every single test needs the exact same execution environment. There is a tendency over time to continue to pile things into SetUp and TearDown that are only needed for a few tests. This has the potential to impact readability and performance.
4. Inadvertant Information Leaking
Some test frameworks (notably NUnit) may create test classes once, and run all the tests using that single instance of the object. If you're not careful, you may inadvertantly leak information across test methods.
Some Solutions
My preferred solution is simply to avoid SetUp and TearDown. If the work done in SetUp and TearDown is simple, move it inline to your test:
[TestFixture]
public class MyTests
{
[Test]
public void FooIsNotNull()
{
using (Foo theFoo = new Foo())
{
Assert.IsNotNull(theFoo);
}
}
}
There's an additional benefit here, which is that you can use the C# using keyword to automate the call to Dispose the object.
If your object creation is complex, you can push that off into a factory method:
[TestFixture]
public class MyTests
{
[Test]
public void FooIsNotNull()
{
using (Foo theFoo = CreateXmlConfiguredFoo())
{
Assert.IsNotNull(theFoo);
}
}
private Foo CreateXmlConfiguredFoo()
{
[... complex initialization logic ...]
}
}
You can see everything you need in the test: the call to the factory method, which is easy to follow; the use of a local instead of a member; the cleanup of the object with the using block. This also leaves you the option of writing different factory methods for different types of initialization.
My Ask
If I could get just one feature into the predominant test runners, it would be to randomize the execution order of test methods. This would help uncover the inadvertant test information leaking problem by highlighting that tests should not be order dependent.