Unit Tests with .NET Core and VSTS

Executing unit tests as part of a Visual Studio Team Services (VSTS) build process has, until now, been the most pain free process I’ve ever encountered. In fact, if you select the Visual Studio build template, there is already a step that handles executing and publishing results of your unit tests. It assumes the convention that your DLL will have the word ‘test’ somewhere in the name. Even if it doesn’t, it is configurable, so you can come up with your own wild-card expression that will work for you.

I have taken for granted that executing these tests in VSTS just works. That is until I went to configure my first CI build for my .NET Core solution. I looked at the output of the build, and while it did not error, it also did not find any tests to run. It did not take me long to realize that, as-is, the step does not execute the unit tests.
Getting my unit tests to work locally wasn’t exactly a walk in the park, but after a couple of searches, I had it working in about 10 – 15 minutes (this is out of scope for this article). All in all, not too bad.
I have found two ways to get the unit tests execute AND the results published as part of the build output.

Visual Studio Test

This is the same step that has worked for me in the past. I was able, through a handful of searches, to discover what settings needed to be changed in order for this step to continue working as expected.
The first setting is Test Assembly. Typically this points to a compiled DLL, but for .NET Core, it needs to point to the project.json files, so I updated the value to **\test\**\project.json. In my project, I am following the convention that all unit test projects are located under a “test” folder in the root of the solution, you may need to change yours. For example, if your test projects are mixed in with the non-test project, a value of **\*Test*\project.json may be what you need.
The second setting is hidden in the “Advanced Execution Options” section that you have to expand, it is called “Other console options”. Ultimately, this step executes vstest.console.exe, so this is where any additional command line arguments are placed. In our case, we want to set the value to /UseVsixExtensions:true /logger:trx. This tells the application to use VSIX extensions that are installed. Fortunately, at the very least, the xUnit VSIX is installed on the build server, so it worked for me.
Below are what my settings look like:

Script dotnet test

The IT Leader's Guide to Multicloud Readiness

This guide provides practical key insights and important factors to consider to make informed decisions in your multicloud journey.

Download the Guide

This is actually the first solution I found. If you decide not to use the Visual Studio Test step, there’s nothing to stop you from scripting it out yourself. I was able to get the unit tests running quickly using the following PowerShell command:

gci test\project.json -Recurse | % {dotnet test $_}

This iterates over all the project.json files located somewhere under the test folder, and executes dotnet test, which uses the settings in the project.json file to determine how to execute the unit tests. After checking in this file, updating the build definition, and executing the build, the unit tests were all executed; there was just one problem: the results did not show up in the build summary screen.
Not surprisingly, VSTS doesn’t parse the output to get all the unit tests information (that would be too fragile and would likely break whenever the specific unit test provider decided to change their output). Fortunately VSTS provides a Publish Test Results step. I updated my PowerShell command:

gci test\project.json -Recurse | % {dotnet test $_ -xml "$($_.DirectoryName)\Test-Results.xml"}

Each provider has a similar switch for specifying an output file. The above works for xUnit, but you may need to alter it to get to work with the test suite you are using.
I executed the updated command and verified that it was writing the results to disk. Now I needed to configure the publish results test to read these files. This was fairly simple:

Note: For the sake of brevity of this post, I left out the fact that I started with NUnit. It was at this point that I switched to xUnit. The best I can tell is that the Publish Test Results Task’s NUnit format is for NUnit 2.0, but you must use NUnit 3.0 for .NET Core (which has a different results format). This left me at an impasse and I had to switch over to xUnit. I hope VSTS build servers get updated soon to work with NUnit 3.0 format, but so far, my experience with xUnit has been good, so I will likely not switch back for this project.

With these changes in place, I ran the build, and not only did it execute the unit tests as expected, but the results were published successfully and are view-able from the build summary screen.
I like the “out of the box”-ness of the first solution (plus, any one who comes behind looking at the build definition will quickly realize the purpose of the steps and probably not look into the details); however, “fool me once, shame on you…” comes to mind, so I feel the second solution is less likely to give me issues in the future. Ultimately, I will likely go with the second solution as it has the fewest prerequisites for the build server. As long as the build server has dotnet.exe installed (which is a must as it is needed to compile the code itself), then all the files needed to execute the unit tests will be downloaded as part of the dotnet restore command.
Hopefully this will save you, the reader, time and headache while you are setting up your builds!