This article is about "improvement" in Visual Studio 2012 unit testing. I think it doesn't happen very often to have an upgrade which makes the functionality worse than the previous version. In the case of the new Test Explorer window we can even talk about removing previously available functionality because the new window is mostly useless for any bigger solution. Until Microsoft improves the window we are not away from the testing experience known before Visual Studio 2008 when we didn't have any built in support for unit testing and we had to use third party plugins or external test runners.
I wanted to write this article immediately after I tried the new Test Explorer in Visual Studio 2012 Ultimate RC but I still believed that Microsoft will improve it before launching Visual Studio 2012 especially when complains to the new Test Explorer appeared long time before RC. Unfortunately Visual Studio 2012 RTW was released with the same Test Explorer window and there is also no update available yet. Test Explorer window should be hopefully improved this fall. There is also related request on Visual Studio User Voice with few other suggestions for improvements in comments.
The rest of the article describes what is wrong with the new window and compares the functionality with Visual Studio 2010 and Resharper 7.
Unit testing improvements in Visual Studio 2012
Visual Studio 2012 improves the built-in support for unit testing by offering extensibility for third party or open source testing frameworks. This is a great improvement because it allows us installing small extension from Visual Studio Online Gallery and integrate for example NUnit or xUnit with Visual Studio Unit Test Platform without installing any other fat plugin (separate test runners). Unfortunately this improvement is at the moment more a theory for two reasons:
- The integration between third party testing framework and Visual Studio is still a plugin and because of that it doesn't work with Visual Studio 2012 Express editions. So the new availability of unit testing in Visual Studio 2012 Express doesn't change anything we did before - we will still use NUnit, MbUnit or xUnit and run them with external test runners. The only exception is at the moment Visual Studio 2012 Express for Windows 8 where test library has to be tested as Windows Store app (it must run as WinRT) and it will not be possible to test it with current external test runners.
- The new Test Explorer makes this very nice feature in simple demonstrations but it becomes quite a mess when we start using it in the real world project - I will demonstrate it with Entity Framework source code which has almost 7000 tests.
Have a fun with the new Test Explorer
Visual Studio 2012 offers only single window to handle, run and manage unit tests and all other types of tests written with common testing frameworks. This window is called Test Explorer:
The screenshot is from Visual Studio 2012 Professional but higher editions only add a window for code coverage. There would be nothing wrong with a single window if it would work as expected and if it would be easy and natural to use. So lets check what we can see if we use the window:
The screenshot contains small subset of Entity Framework unit tests. So what exactly is wrong with the window?
- The window is trying to combine both test runner and test browser and it really doesn't work when it comes to solutions with lot of tests or with different types of tests.
- We cannot remove or add tests to the explorer manually. The explorer always contains all tests from the solution. The only way to control set of tests we want to run is by selection, filtering or few commands to run current test, all tests in scope or all tests in namespace.
- By default there is no filter specified so running tests will always run all of them. Any filtering is done through typing some magic filter into the text box on the top. We can either:
- Type directly - filters tests by their names.
- Use FullName:"SearchTerm" to filter tests by class names or namespaces - the example shows my attempt to get only tests from System.ComponentModel.DataAnnotations namespace. I expected only 12 tests but the result shows 30 because the evaluation of the filter also includes all nested namespaces.
- Use FilePath:"SearchTerm" to filter tests by file paths or names
- Use Result:"SearchTerm" to filter tests by their result. Why do I need to type anything to filter by result? I thought it is an enumeration so there can be some buttons or drop down.
- Filters are evaluated like String.Contains (except the filter on Result where exact case insensitive match is used) and it looks like they don't accept wildcards nor regular expressions.
- There is no autocompletion or IntelliSense when typing the filter.
- I haven't found any way to build complex search criteria. I saw somewhere screenshot with | character used but it didn't work for me. The documentation is also very bare.
- Being able to run tests only from few assemblies looks impossible without manually selecting them in the explorer. Even running tests from single assembly looks like a challenge because there is no command in the context menu for that. Shortcuts for running tests in the same scope or in the same namespace also don't cover this scenario because they don't work from Solution Explorer.
- Filtering tests by categories is not possible.
- There are absolutely no useful groupings. No grouping by project/namespace/class. No grouping by test category/trait (categories are completely ignored by the new explorer). The only available groupings are duration and outcome.
- It is quite hard to select correct tests when they have duplicate names without grouping by the class. When I started using the new Test Explorer I even switched to the naming convention where every test name started with the name of the class under test but it failed immediately when I used inheritance in tests classes.
- It is not visible on the screenshot but another missing feature is support for data driven tests where test method receives parameters from some external source (attribute, another method or even file/excel/database) and every evaluated set of parameters should be considered as a separate test. The most probable reason why this is not supported is very bad support for data driven tests in MS Test framework (it supports only external data sources accessible through database providers) and no support for data driven tests in MS Test framework for Windows Store apps because these apps cannot access databases.
This implementation may be good for smaller solutions where we have only a smaller set of unit tests. In such scenario it may not be a problem to run all unit tests every time but once we have even single more complicated integration/system/performance test written in a testing framework we need to have a better control over the selection of tests to run. That is the scenario where current Test Explorer completely fails because we need manually select tests, classes or assemblies which should be run by the test runner and it is at the moment too complex or maybe even impossible to achieve that through the filter. There is probably also no possibility to exclude some tests from this auto magic running without excluding the whole project from the solution.
Let's check how the new Test Explorer works with Entity Framework source code. Entity Framework has tests divided into three assemblies: unit tests (3300+ tests), functional tests (3500+ tests) and special VB tests (less than 50 tests). The second group represents integration tests working with the real database - those are tests I really don't want to run every time because they are usually 50-100x slower than unit tests. If I run just all 3300+ unit tests on my machine it takes more than 3 minutes to complete (Windows Experience Index of my machine is 7,4). I thought the new Test Explorer was developed to support agile teams in both test first and code first development but I really wonder how could anybody use TDD if he has to wait 3 minutes to complete tests after each small change to code. This approach kills productivity. Just few minutes with Entity Framework source code produced questions like:
- How are projects with lot of unit tests supposed to work with the new Test Explorer?
- What is supposed workflow to use the new Test Explorer with projects containing integration and long running tests?
- How can we manage tests we want to run during development without excluding other projects from solution?
- How can we exclude some tests from the run if they are members of the same assembly as test we want to run?
- How can we work on isolated feature with running only selected tests related to the feature?
All these questions become even more important when we try to use Run Tests After Every Build feature available in Visual Studio 2012 Ultimate.
Resharper 7 - less magic = more control
Resharper doesn't try to combine test runner and test browser. It still contains two separate windows. The Unit Test Explorer window in Resharper shows all tests in the solution in the structured way (defined by grouping):
The screenshot shows the clear visual representation of the test hierarchy defined by the selected grouping. We can just select what tests, test classes, namespaces, assemblies or categories we are going to use and start a new test session or append them to an existing test session. All active sessions are available as tabs in Unit Test Session window:
All tests are again presented in the clear hierarchy defined by the selected grouping. If any of those tests would be data driven it would show one more level in the tree with all test cases passed to the test. As described in the previous link we can also globally define test categories which are ignored when running all tests in Visual Studio to avoid some long running tests or other groups we don't want to deal with.
Even Resharper is not perfect. It probably doesn't have filtering in tests - or at least I don't know about it because with its current implementation I have never needed to search for tests. It's extensibility model is probably based on features provided by NUnit and for example grouping by categories with xUnit doesn't work because xUnit doesn't have special attribute for specifying category - it uses more generic traits instead. It may be issue of integration between Resharper and xUnit but the result is that I don't know how to group tests by traits.
Resharper is also much more slower when it comes to running a lot of tests. Test runs for Entity Framework did not differ too much but in case of ASP.NET Web Stack, Resharper needs 5 minutes to run all tests whereas Visual Studio needs only 2 minutes to run all tests (all on my computer). I'm not sure where is the difference but Test Explorer sees only about 8300 tests but Resharper sees about 200 more and when it expands all data driven tests the total number of tests grows to more than 13000 tests.
It may seem unfair to compare Resharper with Visual Studio because Resharper is separate commercial product. Yes, personal license of Resharper C# Edition costs $149 and license for companies and organizations costs $249 per developer but it is not just a test runner - it is a productivity tool. Visual Studio is also commercial product and if we talk about Premium or Ultimate editions they are damn expensive and they still offer Test Explorer window which cannot compete by its functionality even with free external test runners shipped with open source testing frameworks.
Do you remember Visual Studio 2010?
Visual Studio 2010 didn't provide extensible testing platform. All testing tools for developers were tightly coupled with MS Test framework but even with this limitation it still provided much better test management. We had List View window to browse and selectively execute available tests. This view had limited options to group tests but also options to add additional columns to the view:
We were also able to filter view by different data including test categories:
Executed tests were showed in the separate Test Result window again with options to group tests, filter tests and show additional information:
Unit testing tools in Visual Studio 2010 were bad. They were too complex and hard to use and I always switched to other non-Microsoft solution like NUnit or xUnit run by external test runner or by Resharper when it was possible. The new version of Visual Studio really needed some refactoring and simplification of unit testing tool set but the current version is simplified so much that it lacks core features to be useful.
The new dialog is not a problem for developers who didn't used MS Test framework or MS testing features in Visual Studio 2010. These developers can upgrade to Visual Studio 2012 and by using their current tool set they will be unaffected by this change. Developers who used only features available in Visual Studio 2010 will not enjoy the new "improved" version. They will miss key features to manage their test suite and to control which tests should be run. It is not fun when you pay for the upgrade and get worse tool.
I would really like to see use cases / user stories used to define the new Test Explorer. I would also like to know how people who are behind this feature test their code, how Microsoft's QA tested this feature and how teams in Microsoft use this feature (I wonder if they use it at all)? It could explain a lot of questions but it could still relate mostly to Release candidate. The fact that this feature moved probably without any change from Developer Preview or Beta to official Launch is not a failure of developers. It is even not a failure of analyst / product owner. It looks like a problem of insufficient resources to fix / improve the window on time. In such case it is a management failure.