Test Coverage Analysis with Coverlet in .NET
Test coverage is a metric that quantifies the proportion of a program’s source code exercised by a set of test cases. It is typically expressed as a percentage and is an important indicator of the effectiveness of testing in validating the software.
There are several types of test coverage, including:
Line Coverage
Calculates the percentage of code lines that have been executed by the test cases. Each line of code is marked as covered (executed) or uncovered (not executed).
Branch Coverage
Measures the percentage of conditional execution paths that have been tested. This involves ensuring that all decision branches (such as if/else statements) have been evaluated in all possible states.
Function Coverage
Evaluates how many functions or methods in the code have been called by the tests. Each function is marked as covered if it is called during test execution.
Benefits of Test Coverage Analysis
It provides information about which parts of the code have been exercised by the tests and which parts have not. Test coverage serves several important purposes:
Assess Test Quality
Test coverage helps assess the effectiveness of automated tests. High test coverage typically indicates that most parts of the code have been tested, which is a positive sign of test quality. However, low coverage may indicate areas of the code that are not adequately tested.
Identify Untested Code
Test coverage identifies parts of the code that have not been exercised by any tests. This can help identify potentially problematic or overlooked areas in the code.
Improve Code Quality
Test coverage analysis can highlight redundant, unreachable, or unused code. This can lead to code improvements, such as the removal of dead code.
Prioritize Additional Tests
Based on test coverage, you can prioritize areas of the code that need additional testing. This helps direct testing efforts toward the most critical or complex parts of the code.
Monitor Code Changes
By regularly running test coverage, you can monitor how code changes affect coverage. This helps ensure that new features or bug fixes do not introduce regressions in untested areas.
Compliance Standards
In some cases, compliance standards or industry regulations may require a certain level of test coverage to ensure software quality and safety.
Confidence and Error Reduction
Good test coverage provides greater confidence that the software will work as expected and helps reduce the occurrence of undetected errors.
It’s important to note that while test coverage is a valuable metric, high test coverage does not guarantee that all necessary test cases have been considered, nor does it ensure the quality of the tests themselves. Therefore, it’s essential to consider test quality alongside coverage to ensure the effectiveness of the testing process.
.NET Example Project Setups
Here, we will have our example project configured to facilitate the understanding of Coverlet usage.
To enhance comprehension, the project will feature a simple Employee registration process. We will adopt a more “traditional” approach by creating multiple classes with distinct responsibilities in this process.
Folder's organization
Coverlet Framework
Coverlet is a cross platform code coverage framework for .NET, with support for line, branch and method coverage. It works with .NET Framework on Windows and .NET Core on all supported platforms.
Installation
You MUST add package only to test projects and if you create xunit test projects (dotnet new xunit) you'll find the reference already present in csproj file because Coverlet is the default coverage tool for every .NET Core and >= .NET 5 applications, you've only to update to last version if needed
To begin the installation process, it’s ideal for you to be in the tests folder. In our example, the folder is Employee.API.UnitTests.
Install Coverlet Collector
dotnet add package coverlet.collectorThe command is used in the .NET development ecosystem to add a package called coverlet.collector to a project. Here is a more detailed explanation of each part:
dotnet
This is the main command-line tool of the .NET CLI (Command-Line Interface). It is used to perform various tasks related to .NET application development, such as package management, compilation, execution, and more.
add package
This part of the command instructs the .NET CLI to add a NuGet package to the current project. NuGet packages are reusable code libraries that can be used in .NET projects to add additional functionality.
coverlet.collector
This is the name of the NuGet package you want to add to the project. In this case, “coverlet.collector” is used to enable code coverage, which is a technique used to measure how well the source code of a program is tested through automated tests.
Install Coverlet MSBuild
Coverlet also integrates with the build system to run code coverage after tests. Enabling code coverage is as simple as setting the CollectCoverage property to true
dotnet add package coverlet.msbuildcoverlet.msbuild
This is the name of the NuGet package that you want to add to the project. In this case, “coverlet.msbuild”
Using Test with Coverlet
dotnet test /p:CollectCoverage=trueThe command is used in the .NET development environment to run tests with code coverage enabled. Here’s a breakdown of each part of the command:
dotnet
This is the primary command of the .NET CLI (Command-Line Interface). It is used for various tasks related to .NET application development.
test
This is a subcommand of the `dotnet` CLI, used to execute unit tests in a .NET project.
/p:CollectCoverage=true
After the command is run, a coverage.json file containing the results will be published to the test directory (test/Employee.API.UnitTest) as an attachment.
Caso sempre que os testes forem executados queira saber o coverage basta adicionar as seguintes propriedades no arquivo .csproj do seu projeto de teste:
<PropertyGroup>
(others properties...)
<CollectCoverage>true</CollectCoverage>
<CoverletOutputFormat>lcov</CoverletOutputFormat>
<CoverletOutput>./lcov.info</CoverletOutput>
</PropertyGroup>This configuration tells Coverlet to collect coverage information in the “lcov” format and save the result to a file named “lcov.info.”
Test Coverage Result
The result is displayed in the command prompt:
— Line coverage indicates that 72% of the code lines were exercised by the tests. This means that 72% of the program’s code lines were executed during automated test execution.
— Branch coverage indicates that 100% of the code branches were exercised by the tests. This means that all logical decisions in the code, such as conditional statements (if/else), were tested in all their possible branches.
— Method coverage indicates that 90% of the code methods were exercised by the tests. This means that 90% of the program’s methods were called and tested during automated test execution.
Using Coverlet Collection
dotnet test --collect:"XPlat Code Coverage"The command is used to run tests in a .NET project while collecting code coverage information using the XPlat Code Coverage tool. Here’s a detailed explanation of each part of the command:
dotnet test
This is the primary command for running unit tests in .NET projects. It initiates the execution of all unit tests configured in the project.
— collect:”XPlat Code Coverage”
This is an additional argument that you pass to the `dotnet test` command. It instructs the .NET CLI to collect code coverage information using the “XPlat Code Coverage” tool.
What what happens when executing the command?
— The .NET CLI initiates the execution of all unit tests configured in the project.
— During the execution of tests, the “XPlat Code Coverage” tool tracks which parts of the source code are being executed.
— After the tests are completed, the tool create a folder named TestResults inside Employee.API.UnitTest folder with file coverage.cobertura.xml that contains a code coverage report indicating the percentages of the source code that were executed during the tests.
We can also generate this result file with other file formats:
dotnet test --collect:"XPlat Code Coverage;Format=json,lcov,cobertura"Understand coverage.cobertura.xml File
The information within the “coverage.cobertura.xml” file typically includes details about code coverage by class, method, and code line. This allows you to assess which parts of the source code have been tested and which have not.
The exact structure of the file may vary depending on the version of Coverlet you are using, but it typically includes information such as:
- Coverage by class: The percentage of code coverage for each class in the project.
- Coverage by method: The percentage of code coverage for each method within a class.
- Line-by-line coverage: Detailed information about code coverage at the line level, indicating which lines were covered by tests and which were not.
You can use this coverage report file to identify areas of your code that have not been adequately tested and take steps to improve the quality and reliability of your software by adding tests where needed. The exact structure of the file can be found in the Coverlet documentation or through specific code coverage visualization tools for .NET.
In our case, the coverage rate remains at 0% because we haven’t conducted any form of coverage testing.
<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="1" version="1.9" timestamp="1695413251" lines-covered="0" lines-valid="22" branches-covered="0" branches-valid="0">
<sources>
<source>your_root/src/Employee.API/</source>
</sources>
<packages>
<package name="Employee.API" line-rate="0" branch-rate="1" complexity="10">
<classes>
<class name="Employee" filename="Employee.cs" line-rate="0" branch-rate="1" complexity="3">
<methods>
<method name="get_Id" signature="()" line-rate="0" branch-rate="1" complexity="1">
<lines>
<line number="3" hits="0" branch="False" />
</lines>
</method>
<method name="get_Name" signature="()" line-rate="0" branch-rate="1" complexity="1">
<lines>
<line number="4" hits="0" branch="False" />
</lines>
</method>
<method name="get_Document" signature="()" line-rate="0" branch-rate="1" complexity="1">
<lines>
<line number="5" hits="0" branch="False" />
</lines>
</method>
</methods>
<lines>
<line number="3" hits="0" branch="False" />
<line number="4" hits="0" branch="False" />
<line number="5" hits="0" branch="False" />
</lines>
</class>
<!-- Others classes... -->
</classes>
</package>
</packages>
</coverage>Report Generator
ReportGenerator converts coverage reports generated by coverlet into human readable reports in various formats.
The reports show the coverage quotas and also visualize which lines of your source code have been covered.
ReportGenerator supports merging several coverage files into a single report.
Installation
To begin the installation process, it’s ideal for you to be in the tests folder. In our example, the folder is Employee.API.UnitTests.
To install, we’ll need to execute several commands. Below are each of the commands along with their descriptions.
Create Global Tool
dotnet tool install -g dotnet-reportgenerator-globaltoolThe command is used in the .NET Core development ecosystem to install a global tool named dotnet-reportgenerator-globaltool. Let me break down each part of the command for you:
dotnet
This is the primary command in the .NET Core Command-Line Interface (CLI). It is used to perform various tasks related to .NET application development, including package and tool management.
tool install
This part of the command indicates that you are using the command to install a global tool in your development environment.
-g
This option is used to specify that the tool is being installed globally. This means that the tool will be available for use in any project or location on the system, not just in a specific project.
dotnet-reportgenerator-globaltool
This is the name of the tool you are installing. In this case, “dotnet-reportgenerator-globaltool” is a tool used to generate code coverage reports from test outputs in .NET projects.
After running this command, the “dotnet-reportgenerator-globaltool” tool will be installed in your environment and ready for use. You can then use it to create code coverage reports from test outputs in .NET projects, which is useful for assessing the quality of your tests and identifying areas of code that need more test coverage.
Create Tool in Specific Path
Command similar to the previous one with an additional option.
dotnet tool install dotnet-reportgenerator-globaltool --tool-path tools— tool-path tools
This option specifies a custom directory path, in this case, "tools," where the tool should be installed. By using this option, you can control the installation location of the tool. In this example, the tool will be installed in a directory named "tools" within the current working directory.
Create Tool Manifest
dotnet new tool-manifestThis command is used in the .NET Core development environment to create a tool manifest file. The tool manifest file is used to manage global tools in a .NET Core project. Here’s a more detailed explanation:
dotnet
This is the main command in the .NET Core Command-Line Interface (CLI). It’s used for various tasks related to .NET application development.
new
This part of the command indicates that you want to create something new. In this case, you’re creating a tool manifest file.
tool-manifest
This is the option that specifies the type of new item you’re creating, which is a tool manifest file.
When you execute this command in a project directory, it creates a file named “dotnet-tools.json” in the project’s root directory. This tool manifest file is used to list and manage the global tools that the project depends on. Global tools are utilities that can be used system-wide and aren’t tied to a specific project.
You can manually edit the “dotnet-tools.json” file to add the global tools that your project needs. Then, you can use the `dotnet restore` command to restore these global tools into your development environment, making them available for use in your project.
This tool manifest file is useful when you want to maintain precise control over the tools used in your .NET Core project and ensure that everyone working on the project has access to the same global tools.
Create Tool in the new Manifest
dotnet tool install dotnet-reportgenerator-globaltoolEssentially, it’s the same command we used earlier, but without being installed globally, meaning without the “-g” option. This will configure this tool within the new manifest we just created.
Summary of commands
dotnet new tool-manifest
Creates a tool manifest file in the current project directory to manage global tools within the project.
dotnet tool install -g dotnet-reportgenerator-globaltool
Installs the global .NET Core tool “dotnet-reportgenerator-globaltool” system-wide, making it accessible from any location.
dotnet tool install dotnet-reportgenerator-globaltool — tool-path tools
Installs the “dotnet-reportgenerator-globaltool” tool but specifies a custom directory path (“tools”) for its installation within the project.
dotnet tool install dotnet-reportgenerator-globaltool
Installs the "dotnet-reportgenerator-globaltool" tool without the "-g" option, configuring it within the project.
Using Report
dotnet reportgenerator "-reports:coverage.cobertura.xml" "-targetdir:coveragereport" -reporttypes:HtmlThe command is used to generate a code coverage report using the ReportGenerator tool. Here’s an explanation of each part of the command:
dotnet reportgenerator
This is the command to run the ReportGenerator tool in the .NET environment.
”-reports:coverage.cobertura.xml”
This part of the command specifies the input coverage report file to be used for generating the code coverage report. In this case, the file is named “coverage.cobertura.xml.” The -reports flag is used to specify the report file.
”-targetdir:coveragereport”
This part of the command specifies the target directory where the generated code coverage report will be saved. In this case, the directory is named “coveragereport.” The -targetdir flag is used to specify the target directory.
-reporttypes:Html
This part of the command specifies the type of code coverage report to be generated. In this case, it’s set to “Html,” which means that an HTML-based code coverage report will be generated.
When you run this command, it will take the input coverage report file (“coverage.cobertura.xml”) and generate an HTML code coverage report in the specified target directory (“coveragereport”). The HTML report will contain information about code coverage, such as which lines of code were covered by tests and which were not, making it easier to visualize and understand the test coverage of your codebase.
Report Result
After executing the command, you will see in the prompt that the file has been generated at coveragereport/index.html.
Opening the file will allow us to view a test coverage report:
Conclusion
In conclusion, test coverage is a crucial metric in software development that quantifies the extent to which a program’s source code is tested by a set of test cases. It provides insights into the effectiveness of testing efforts and helps identify areas of code that require additional testing.
This article explored the benefits of test coverage, including its role in assessing test quality, identifying untested code, improving code quality, prioritizing additional tests, and ensuring compliance with standards. It also emphasized the importance of considering test quality alongside coverage for an effective testing process.
Furthermore, we delved into setting up a .NET example project to demonstrate the practical use of Coverlet for code coverage analysis. We configured the project, installed the necessary packages, and executed tests with code coverage enabled. We also discussed the generation of code coverage reports using the ReportGenerator tool.
Understanding and effectively utilizing test coverage is essential for building robust and reliable software. By following the practices and tools outlined in this article, developers can gain better insights into their code’s quality and take proactive steps to enhance it.
