Reporting¶
SpecFlow+ LivingDoc¶
Reporting is now easier to generate and share thanks to SpecFlow+ LivingDoc.
You can see your test execution results in a dynamic way now and also check for any unused step definitions along with many other handy features that truly bring your reporting and documentation to life!
Head over to SpecFlow+ LivingDoc to read more.
Here is a quick snapshot:
Reporting prior to SpecFlow 3¶
These reports are only available prior to SpecFlow 3! They have been removed in SpecFlow 3.
SpecFlow provides various options to generate reports related to the acceptance tests.
Note: The specflow.exe command line tool that is used to generate reports can be found in the packages\Specflow.{version number}\tools directory, when you installed SpecFlow through NuGet. Start the tool with no parameters or use the --help option to display an overview of the available options.
You can find a repository containing the old report code here. For information on why the reports were moved to a seperate repo, please read see this GitHub issue.
Test Execution Report¶
This report generates an HTML test execution report. The report contains a summary of the executed tests and their results, as well as a detailed report of the execution of individual scenarios.
The following sub-sections cover generating the test execution report for different unit test providers.
NUnit Test Execution Report¶
NUnit 2¶
In order to generate this report, execute the acceptance tests with the nunit-console runner. This tool generates an XML summary of the test executions. To include detailed scenario execution traces, you need to capture the test output using the /out and the /labels options, e.g.
nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll
NUnit 3¶
In order to generate this report, execute the acceptance tests with the nunit3-console runner, and set it to output the results in nunit2 format. To include detailed scenario execution traces, you need to capture the test output using the --out and the --labels=All options (see the example below).
Important: The NUnit.Extension.NUnitV2ResultWriter package must be included in your project, otherwise you will receive the message: “Unknown result format: nunit2”.
nunit3-console.exe --labels=All --out=TestResult.txt "--result=TestResult.xml;format=nunit2" bin\Debug\BookShop.AcceptanceTests.dll
SpecFlow Report Generator¶
Note: The examples and parameters are for version 2.4.* and higher. Older versions can be found by viewing past revisions in the GitHub wiki.
The report generation step is the same for both versions of NUnit. The two generated files can be used to invoke the SpecFlow report generation.
If you use the default output file names shown above, you only need to specify information about the C# test project containing the *.feature files. Specflow uses the default TestResult.xml and TestResult.txt files to produce TestResult.html.
specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj
specflow.exe nunitexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj --xmlTestResult CustomNunitTestResult.xml --testOutput CustomNunitTestOutput.txt --OutputFile CustomSpecflowTestReport.html
The following table contains the possible arguments for this command.
| Attribute | Value | Description |
|---|---|---|
| -p, --ProjectFile | A path of the project file containing the `*.feature` files. Required. | |
| -t, --xmlTestResult | NUnit XML test result file | The XML test result file generated by nunit-console. Optional. Default: TestResult.xml |
| --testOutput | NUnit test output file | The labeled test output file generated by nunit-console. Optional. Default: TestResult.txt |
| --OutputFile | HTML file | Generated Output File. Optional. Default: TestResult.html |
| -x, --XsltFile | XSLT file | Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional. Default: not specified |
| -n, --ProjectName | Project name | Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional. Default: not specified |
| -l, --FeatureLanguage | Feature language | Default: en-US |
MsTest Test Execution Report¶
``` specflow.exe mstestexecutionreport --ProjectFile BookShop.AcceptanceTests.csproj /testResult:result.trx /out:MyResult.html ```
The following table contains the possible arguments for this command.
| Attribute | Value | Description |
|---|---|---|
| -p, --ProjectFile | A path of the project file containing the feature files. This is a mandatory argument. | |
| -t, --testResult | MsTest test result (trx) file | The TRX test result file generated by MsTest. Optional. Default: TestResult.trx |
| -n, --ProjectName | Project name | Project name which can be passed explicitly instead of implicitly getting it from --ProjectFile. Optional. Default: not specified |
| -o, --OutputFile | HTML file | Generated Output File. Optional. Default: TestResult.html |
| -l, --FeatureLanguage | Feature language | Default: en-US |
| -x, --XsltFile | XSLT file | Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional. Default: not specified |
Step Definition Report¶
This report shows the usage and binding status of the steps in your entire project. You can use this report to find both unused code in the automation layer and scenario steps with no binding defined.
Steps with a red background are steps that exist in the automation layer but are not used in any feature files.
Steps with a yellow background are steps that exist in a feature file but do not have a corresponding binding defined.
Steps without a special backgrounds are steps that exist both in feature files and the automation layer. Ideally, all your steps are like this.
specflow.exe stepdefinitionreport --ProjectFile BookShop.AcceptanceTests.csproj /BinFolder:bin/debug
The following table contains the possible arguments for this command.
| Attribute | Value | Description |
|---|---|---|
| projectFile | A path of the project file containing the feature files. This is a mandatory argument. | |
| /binFolder | A folder path (absolute or relative to the project folder) | A path for the compiled SpecFlow project. Optional. Default: bin\Debug |
| /out | HTML file | Generated Output File. Optional. Default: StepDefinitionReport.html |
| /xsltFile | XSLT file | Custom XSLT file to use, defaults to built-in stylesheet if not provided. Optional. Default: not specified |