Share via

That Pesky MSTest Execution Ordering..

  • Article

I have come upon several groups who are puzzled by the intricacies of execution of the MSTest Framework. I admit - it is quite confusing, so I hope that these postings will help to clean some of the confusion.

I think that most confusion comes from some user’s expectation of MSTest to execute like the Nunit framework. They execute differently since Nunit instantiates a test class only once when executing all the tests contained in it, whereas MSTest instantiates each test method’s class separately during the execution process, with each instantiation occurring on a separate thread. This design affects 3 specific things which often confuse users of MSTest:

1. ClassInitialize and ClassCleanup: Since ClassInitialize and ClassCleanUp are static, they are only executed once even though several instances of a test class can be created by MSTest. ClassInitialize executes in the instance of the test class corresponding to the first test method in the test class. Similarly, MSTest executes ClassCleanUp in the instance of the test class corresponding to the last test method in the test class.

2. Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees regarding the order of execution of unit tests in a single class, or across classes. The execution of tests may be interleaved across classes, and potentially even assemblies, depending on how you chose to execute your tests. The key thing here is – all tests could be executed in any order, it is totally undefined.

3. TextContext Instances: TestContexts are different for each test method, with no sharing between test methods.

For example, if we have a Test Class:

[TestClass]

public class VSTSClass1

{

private TestContext testContextInstance;

public TestContext TestContext

{

get

{

return testContextInstance;

}

set

{

testContextInstance = value;

}

}

[ClassInitialize]

public static void ClassSetup(TestContext a)

{

Console.WriteLine("Class Setup");

}

[TestInitialize]

public void TestInit()

{

Console.WriteLine("Test Init");

}

[TestMethod]

public void Test1()

{

Console.WriteLine("Test1");

}

[TestMethod]

public void Test2()

{

Console.WriteLine("Test2");

}

[TestMethod]

public void Test3()

{

Console.WriteLine("Test3");

}

[TestCleanup]

public void TestCleanUp()

{

Console.WriteLine("TestCleanUp");

}

[ClassCleanup]

public static void ClassCleanUp()

{

Console.WriteLine("ClassCleanUp");

}

}

(This consists of 3 Test Methods, ClassInitialize, ClassCleanup, TestInitialize, TestCleanUp and anexplicit declaration of TestContext)

The execution order would be as follows:

Test1 [Thread 1]: new TestContext -> ClassInitialize -> TestInitialize -> TestMethod1 -> TestCleanUp

Test2 [Thread 2]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp

Test3 [Thread 3]: new TestContext -> TestInitialize -> TestMethod2 -> TestCleanUp -> ClassCleanUp

The output after running all the tests in the class would be:

Class Setup

Test Init

Test1

TestCleanUp

Test Init

Test2

TestCleanUp

Test Init

Test3

TestCleanUp

ClassCleanUp

 (A special thanks to Dominic Hopton for his edits of my hackery on this post :))

Comments

  • Anonymous
    April 01, 2007
    The comment has been removed
  • Anonymous
    July 18, 2007
    The comment has been removed
  • Anonymous
    January 20, 2008
    I extended the test a little bit to see how MSTest behaves with VS2005 and multiple test classes in different test projects. I also added code to print on what thread the methods get called. This is what I see when I run the tests from the Test View (please note the nice typo :)

Class Setup 1 (on thread AdpaterExeMgrThread1) Test Init 1 (on thread AdpaterExeMgrThread1) Test 1.1 (on thread AdpaterExeMgrThread1) TestCleanUp 1 (on thread AdpaterExeMgrThread1) Test Init 1 (on thread AdpaterExeMgrThread1) Test 1.2 (on thread AdpaterExeMgrThread1) TestCleanUp 1 (on thread AdpaterExeMgrThread1) Test Init 1 (on thread AdpaterExeMgrThread1) Test 1.3 (on thread AdpaterExeMgrThread1) TestCleanUp 1 (on thread AdpaterExeMgrThread1) Class Setup 2 (on thread AdpaterExeMgrThread1) Test Init 2 (on thread AdpaterExeMgrThread1) Test 2.1 (on thread AdpaterExeMgrThread1) TestCleanUp 2 (on thread AdpaterExeMgrThread1) Test Init 2 (on thread AdpaterExeMgrThread1) Test 2.2 (on thread AdpaterExeMgrThread1) TestCleanUp 2 (on thread AdpaterExeMgrThread1) Test Init 2 (on thread AdpaterExeMgrThread1) Test 2.3 (on thread AdpaterExeMgrThread1) TestCleanUp 2 (on thread AdpaterExeMgrThread1) ClassCleanUp 1 (on thread TestCasemanager::ExecutionThread) ClassCleanUp 2 (on thread TestCasemanager::ExecutionThread)

So all tests are run on the same thread and only the cleanup routines are called from a different thread. The class clean up methods are postponed until all test methods from all test classes have been run, which in my opinion makes the class initialize / clean up methods pretty useless unless your unit tests are very cleanly isolated.

  • Anonymous
    January 20, 2008
    Hmm, it seems that MsTest just does not name the threads - using the ManagedThreadId property we see something else: Class Setup 1 (on thread 17) Test Init 1 (on thread 17) Test1.1 (on thread 17) TestCleanUp 1 (on thread 17) Test Init 1 (on thread 19) Test1.2 (on thread 19) TestCleanUp 1 (on thread 19) Test Init 1 (on thread 20) Test1.3 (on thread 20) TestCleanUp 1 (on thread 20) Class Setup 2 (on thread 21) Test Init 2 (on thread 21) Test2.1 (on thread 21) TestCleanUp 2 (on thread 21) Test Init 2 (on thread 19) Test2.2 (on thread 19) TestCleanUp 2 (on thread 19) Test Init 2 (on thread 17) Test2.3 (on thread 17) TestCleanUp 2 (on thread 17) ClassCleanUp 1 (on thread 16) ClassCleanUp 2 (on thread 16)

  • Anonymous
    April 23, 2008
    The comment has been removed

  • Anonymous
    August 14, 2008
    Насколько я помню, в классике модульного тестирования принято, что порядок выполнения модульных тестов

  • Anonymous
    May 01, 2012
    Sorry, but this is mad. I would like to define the tests in one specific order, so if one fails... there is no point in go on testing. It looks like the MS developers do not use their own products in real life scenarios. It is just disappointing.

  • Anonymous
    May 10, 2012
    The comment has been removed

  • Anonymous
    July 24, 2012
    Your unit tests should never abort simply because one failed... That defeats the purpose of unit tests entirely.

  • Anonymous
    October 30, 2013
    Nearly 7 years later, this post is still useful. But it could be improved by adding the test class constructor: public VSTSClass1()    {        Debug.WriteLine("VSTSClass1 Constructor");    } to demonstrate that the constructor is run (surprisingly) once per TestMethod.

  • Anonymous
    April 26, 2015
    8 years later, still handy.

  • Anonymous
    June 19, 2015
    There is also AssemblyInitializeAttribute. It decorates a static method that runs before any test in the assembly have run:        [AssemblyInitialize()]        public static void AssemblyInit(TestContext context)        {            // some code        } A method decorated with AssemblyCleanupAttribute can be used to free resources claimed by the assembly, after all tests have run.

  • Anonymous
    October 08, 2015
    @That is totally a pain The scenario you described is not unit testing. Unit tests should work in any order and regardless of how are grouped. You should get the same result if you run a test individually, or as second in a test suite or as last in another test suite. You can do some of the other test types with MS Test, NUnit, XUnit and others but overally you'd be better off with test runners designed for your test types (i.e. acceptance tests, scenario based UI tests). Typically UI tests are where you'd abort test run in case a step fails, and if you translate this to MS Test/XUnit you'd rather have one test and more asserts. I.e. var session = service.Login(username, password) Assert.IsNotNull(session, "Authentication failed."); var data = service.GetSomeData(session); Assert.IsNotNull(data, "Failed to get data.");

  • Anonymous
    October 08, 2015
    @johanw It would be pointless to run TestInitialize and TestCleanup otherwise than are now. Set up your environment in TestInitialize, run the test, then after test clean up in TestCleanup. Why would you clean up before the test was run? A test what requires advanced database setup is not a unit test.