Skip to content

xunit Configuration Files.md

Latest commit

 

History

History
377 lines (358 loc) · 16.8 KB

xunit Configuration Files.md

File metadata and controls

377 lines (358 loc) · 16.8 KB

Configuration Files

Configuration files can be used to configure xUnit.net on a per test-assembly basis.

In these examples, we tell you to use the file name xunit.runner.json. You can also use <AssemblyName>.xunit.runner.json (where <AssemblyName> is the name of your unit test assembly, without the file extension like .dll or .exe). You should only need to use this longer name format if your unit tests DLLs will all be placed into the same output folder, and you need to disambiguate the various configuration files.

The assembly-specific filename takes precedence over the non-specific filename; there is no merging of values between files.

Adding the configuration file

  1. Add a new JSON file to the root of your test project. Name the file xunit.runner.json. Start with a schema reference so that text editors (like Visual Studio & Visual Studio Code) can provide auto-complete behavior while editing the file:

    {
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json"
}

  2. Tell the build system to copy the file into the build output directory. Edit your .fsproj file and add the following:

    <ItemGroup>
  <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

Supported configuration items

The current schema is online at https://xunit.net/schema/current/xunit.runner.schema.json, which can be set in the JSON file to aid with intellisense from development IDEs like Visual Studio and Visual Studio Code.

Configuration elements are placed inside a top-level object:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "enum-or-string-key": "value1",
  "boolean-key": true,
  "integer-key": 42
}

All values in this table list the minimum runner version that is required to support the configuration value (previous versions will ignore unknown configuration values). Features that require support from the test framework itself will also list the minimum version of the test framework required for the feature; if you don't see a minimum test framework version, then it means the feature is implemented purely in the runners and works with all test framework versions.

Key Supported Values appDomain
Runners v2 2.1+
Test Framework v1-v2

Set this value to determine whether App Domains are used. By default, they will be used when available (the ifAvailable value). If you set this to required, it will require that app domains are available; if you set this to denied, it will not use app domains. Note that App Domains are only supported with .NET Framework tests, and only with tests linked against xUnit.net framework v1 or v2.

JSON schema type: enum
Valid values: required, ifAvailable, denied
Default value: ifAvailable

culture
Runners v3+
Test Framework v3+

Set this value to override the default culture used to run all unit tests in the assembly. You can pass default to use the default behavior (use the default culture of the operating system); you can pass invariant to use the invariant culture; or you can pass any string which describes a valid culture on the target operating system (see BCP 47 for a description of how culture names are formatted; note that the list of supported cultures will vary by target operating system).

JSON schema type: string
Default value: default

diagnosticMessages
Runners v2 2.1+
Test Framework v2+

Set this value to true to include diagnostic information during test discovery and execution. Each runner has a unique method of presenting diagnostic messages.

JSON schema type: boolean
Default value: false

failSkips
Runners v2 2.5+
Test Framework v1+

Set this to true to enable skipped tests to be treated as errors.

JSON schema type: boolean
Default value: false

failWarns
Runners v3+
Test Framework v3+

Set this to true to enable warned tests to be treated as errors. By default, warnings will be reported with a passing test result.

JSON schema type: boolean
Default value: false

internalDiagnosticMessages
Runners v2 2.1+
Test Framework v2+

Set this value to true to include internals diagnostic information during test discovery and execution. Each runner has a unique method of presenting diagnostic messages. Internal diagnostics often include information that is only useful when debugging the test framework itself.

JSON schema type: boolean
Default value: false

longRunningTestSeconds
Runners v2 2.2+

Set this value to enable long-running (hung) test detection. When the runner is idle waiting for tests to finished, it will report that fact once the timeout has passed. Use a value of 0 to disable the feature, or a positive integer value to enable the feature (time in seconds).

NOTE: Long running test messages are diagnostic messages. You must enable diagnostic messages in order to see the long running test warnings.

JSON schema type: integer
Default value: 0 (disabled)

maxParallelThreads
Runners v2 2.1+
Test Framework v2+

Set this to override the maximum number of threads to be used when parallelizing tests within this assembly. Use a value of 0 (or default1) to indicate that you would like the default behavior; use a value of -1 (or unlimited1) to indicate that you do not wish to limit the number of threads used for parallelization; use a value with the multiplier syntax1 (i.e., values like 0.5x or 2x) to indicate that you wish you have a multiple of the number of CPU threads (so 2x indicates you want the number of maximum parallel threads to be 2 * the number of CPU threads).

1 NOTE: While this configuration value has been present since v2 2.1, the newer values (default, unlimited, and the multiplier syntax) are only supported with Runners v2 2.8 and later.

JSON schema type: integer (schema v2.1+), integer/string (schema v2.8+)
Default value: the number of CPU threads available to your PC

methodDisplay
Runners v2 2.1+
Test Framework v2+

Set this to override the default display name for test cases. If you set this to method, the display name will be just the method (without the class name); if this set this value to classAndMethod, the default display name will be the fully qualified class name and method name.

JSON schema type: enum
Valid values: method, classAndMethod
Default value: classAndMethod

methodDisplayOptions
Runners v2 2.4+
Test Framework v2+

Set this to automatically perform transforms on default test names. This value can either be all, none, or a comma-separated combination of one or more of the following values:

  • replaceUnderscoreWithSpace Replaces all underscores with spaces
  • useOperatorMonikers Replaces operator names with matching symbols
    • eq becomes =
    • ne becomes !=
    • lt becomes <
    • le becomes <=
    • gt becomes >
    • ge becomes >=
  • useEscapeSequences Replaces escape sequences in the format Xnn or Unnnn with their ASCII or Unicode character equivalents. Examples:
    • X2C becomes ,
    • U1D0C becomes ᴌ
  • replacePeriodWithComma Replaced periods with a comma and a space. This option is typically only useful if methodDisplay is classAndMethod.

JSON schema type: string (comma separated flags)
Valid values: all, none, or comma separated flags
Default value: none

parallelAlgorithm
Runners v2 2.8+
Test Framework v2 2.8+

Set this to change the way tests are scheduled when they're running in parallel. For more information, see Running Tests in Parallel. Note that the algorithm only applies when you have enabled test collection parallelism, and are using a limited number of threads (i.e., not unlimited or -1).

JSON schema type: enum
Valid values: conservative, aggressive
Default value: conservative

parallelizeAssembly
Runners v2 2.1+
Test Framework v2+

Set this to true if this assembly is willing to participate in parallelization with other assemblies. Test runners can use this information to automatically enable parallelization across assemblies if all the assemblies agree to it.

JSON schema type: boolean
Default value: false

parallelizeTestCollections
Runners v2 2.1+
Test Framework v2+

Set this to true if the assembly is willing to run tests inside this assembly in parallel against each other. Tests in the same test collection will be run sequentially against each other, but tests in different test collections will be run in parallel against each other. Set this to false to disable all parallelization within this test assembly.

JSON schema type: boolean
Default value: true

preEnumerateTheories
Runners v2 2.1+
Test Framework v2+

Set this to true to pre-enumerate theories so that there is an individual test case for each theory data row. Set this to false to return a single test case for each theory without pre-enumerating the data ahead of time (this is how xUnit.net v1 used to behave). This is most useful for developers running tests inside Visual Studio, who wish to have the Code Lens test runner icons on their theory methods, since Code Lens does not support multiple tests from a single method.

This value does not have a default, because it's up to each individual test runner to decide what the best default behavior is. The Visual Studio adapter, for example, will default to true, while the console and MSBuild runners will default to false.

JSON schema type: boolean

seed
Runners v3+
Test Framework v3+

Set this to set the seed used for randomization (affects how the test cases are randomized). This is only valid for v3.0+ test assemblies; it will be ignored for v1 or v2 assemblies. If the seed value isn't set, then the system will determine a reasonable seed (and print that seed when running the test assembly, to assist you in reproducing order-dependent failures).

JSON schema type: integer
Valid values: between 0 and 2147483647
Default value: unset

shadowCopy
Runners v2 2.1+
Test Framework v1-v2

Set this to true to use shadow copying when running tests in separate App Domains; set to false to run tests without shadow copying. When running tests without App Domains, this value is ignored.

JSON schema type: boolean
Default value: true

showLiveOutput
Runners v2 2.8.1+
Test Framework v2+

Set this to true to show messages from ITestOutputHelper live during the test run, in addition to showing them after the test has completed; set to false to only show test output messages after the test has completed. Note: when using dotnet test you may need to pass an extra command line option (--logger "console;verbosity=normal") to be able to see messages from xUnit.net, as they are hidden by default.

JSON schema type: boolean
Default value: false

stopOnFail
Runners v2 2.5+
Test Framework v2+

Set this to true to stop running further tests once a test has failed. (Because of the asynchronous nature of test execution, this will not necessarily happen immediately; any test that is already in flight may complete, which may result in multiple test failures reported.)

JSON schema type: boolean
Default value: false