Executing specific Scenarios in your Build pipeline

SpecFlow converts the tags in your feature files to test case categories:

  • SpecFlow+ Runner: TestCategory

  • NUnit: Category or TestCategory

  • MSTest: TestCategory

  • xUnit: Trait (similar functionality, SpecFlow will insert a Trait attribute with Category name)

This category can be used to filter the test execution in your build pipeline. Note that the incorrect filter can lead to no test getting executed.

You don’t have to include the @ prefix in the filter expression.

Learn more about the filters in Microsoft’s official documentation.

Examples

All the examples here are using Category, but if you are using MsTest or SpecFlow+ Runner then you should use TestCategory instead.

How to use the filters

Below are 2 scenarios where one of them has a tag: @done, and the other one does not have a tag.

Feature: Breakfast

@done
Scenario: Eating cucumbers
  Given there are 12 cucumbers
  When I eat 5 cucumbers
  Then I should have 7 cucumbers

Scenario: Use all the sugar
  Given there is some sugar in the cup
  When I put all the sugar to my coffee
  Then the cup is empty

If we would like to run only the scenario with @done tag, then the filter should look like:

Category=done

Below are 2 scenarios where one of them has a tag: @done, and the other one has @automated.

Feature: Breakfast

@done
Scenario: Eating cucumbers
  Given there are 12 cucumbers
  When I eat 5 cucumbers
  Then I should have 7 cucumbers

@automated
Scenario: Use all the sugar
  Given there is some sugar in the cup
  When I put all the sugar to my coffee
  Then the cup is empty

If we would like to run scenarios which have either @done or @automated:

Category=done | Category=automated

Below are 2 scenarios where one of them has a tag: @done, and the other one has @automated. There is also a @US123 tag at Feature level.

@US123
Feature: Breakfast

@done
Scenario: Eating cucumbers
  Given there are 12 cucumbers
  When I eat 5 cucumbers
  Then I should have 7 cucumbers

@automated
Scenario: Use all the sugar
  Given there is some sugar in the cup
  When I put all the sugar to my coffee
  Then the cup is empty

If we would like to run only those scenarios, which have both @US123 and @done:

Category=US123 & Category=done

Below are 2 scenarios where one of them has two tags: @done and @important. There is another scenario, which has the @automated tag, and there is a @us123 tag at Feature level.

@US123
Feature: Breakfast

@done @important
Scenario: Eating cucumbers
  Given there are 12 cucumbers
  When I eat 5 cucumbers
  Then I should have 7 cucumbers

@automated
Scenario: Use all the sugar
  Given there is some sugar in the cup
  When I put all the sugar to my coffee
  Then the cup is empty

If we would like to run only those scenarios, which have both @done and @important:

Category=done & Category=important

dotnet test

Use the --filter command-line option:

dotnet test --filter Category=done

dotnet test --filter "Category=us123 & Category=done"

dotnet test --filter "Category=done | Category=automated"

vstest.console.exe

Use the /TestCaseFilter command-line option:

vstest.console.exe "C:\Temp\BookShop.AcceptanceTests.dll" /TestCaseFilter:"Category=done"

vstest.console.exe "C:\Temp\BookShop.AcceptanceTests.dll" /TestCaseFilter:"Category=us123 & Category=done"

vstest.console.exe "C:\Temp\BookShop.AcceptanceTests.dll" /TestCaseFilter:"Category=done | Category=automated"

Azure DevOps - Visual Studio Test task

>Note: This task is supported only on Windows agents and cannot be used on other platforms.

The filter expression should be provided in the “Test filter criteria” setting in the Visual Studio Test task:

Visual Studio Test task

Visual Studio Test task

Azure DevOps - .NET Core task

Alternatively you could use the dotnet task (DotNetCoreCLI) to run your tests. This works on all kinds of build agents:

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: test
    projects: 'BookShop.AcceptanceTests'
    arguments: '--filter "Category=done"'

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: test
    projects: 'BookShop.AcceptanceTests'
    arguments: '--filter "Category=us123 & Category=done"'