The SpecFlow+ Runner is out of maintenance. Read more about it here.

Execution

The <Execution> element defines how your tests are executed. Use these settings to run tests in parallel by defining multiple threads, determine how often and when tests are retried, and when to abort the test run due to failing tests passing a specific threshold.

You can also schedule tests to run based on your test history stored in the SpecFlow+ Runner server. If you use adaptive mode, previously failing tests are prioritized over tests that have passed.

Attributes

The following optional attributes are available:

stopAfterFailures

  • The number of failed tests after which the execution of all tests is stopped. Set this to 0 (zero) to execute all tests, irrespective of the number of failures. If not specified, this defaults to 10.
Note: The value in the default .srprofile shipped with the NuGet package is 3.

testSchedulingMode

  • Determines the order in which SpecRun executes your tests:
    • Sequential : Tests are always executed in the same order (default). The order in which tests are executed is predictable and always the same, but cannot be changed manually.
    • Random : The order of tests is randomised. This can be useful to detect flickering scenarios that may pass/fail depending on the order in which the other tests area executed.
    • Adaptive : SpecRun uses the statistics available in the pass/fail history to determine which tests to run first. Previously failing tests and new tests are executed before successful and stable tests.
Note: If you set the testSchedulingMode to Adaptive when no previous test execution results are available (e.g. no server has been configured, or the server is unreachable), the tests are executed using the Random option instead.

testThreadCount

  • The number of tests threads used to execute tests default value is 1. While there is no maximum number of threads, setting this value too high can reduce performance as the individual threads compete for processing power.

  • For Process test thread isolation (testThreadIsolation setting), it recommended to set the number of threads to one less that the number of cores available. You can generally use higher numbers for isolation using AppDomain and SharedAppDomains. See Parallel Exeuction documentation for more info on testThreadIsolation.
Note: Just increasing the number of threads on its own may cause issues. E.g. if the threads access the same database. It thus makes sense to combine this with a relocation and transformation of the .config file using the {TestThreadId} placeholder in the name of the new file and database instances.
Note: Before and After hooks (e.g. BeforeTestRun, AfterTestRun etc.) are executed per thread.

retryFor

  • Determines which tests should be retried based on their status:
    • None: No tests are retried.
    • Failing: Failing test are retried (default).
    • All: All tests are retried, including those that pass.
Note: Tests are retried immediately after returning the corresponding status and before any subsequent tests are executed.

retryCount

  • The number of times tests that their execution status matches the value defined in <retryFor> are retried. Default value is 2.

apartmentState

  • Sets the apartment state used to execute the tests:
    • STA : Single Threaded Apartment. Use this if your application is not thread-safe.
    • MTA : Multi-Threaded Apartment.
    • Unknown : The ApartmentState is not set ; tests run in same thread as SpecFlow+. (default).
Note: Setting the apartmentState property does not set the test thread apartment state on Linux and OSX. ApartmentStates are not supported on non-Windows systems because COM interop is only available on Windows. Please refer to Microsoft’s documentation for more information

Example

The following example retries only failed tests regardless of their outcome, using all threads for execution. The order of tests is random in a multi threaded execution and a failed test is re-executed twice:

<Execution retryFor="Failing" stopAfterFailures="0" testThreadCount="Max" testSchedulingMode="Random" retryCount="2" apartmentState="MTA"/>