Migrate from xUnit v2 to xUnit v3 with MTP support#2918
Draft
slang25 wants to merge 11 commits intoApp-vNext:mainfrom
Draft
Migrate from xUnit v2 to xUnit v3 with MTP support#2918slang25 wants to merge 11 commits intoApp-vNext:mainfrom
slang25 wants to merge 11 commits intoApp-vNext:mainfrom
Conversation
|
Member
|
Will Stryker actually get the coverage with MTP yet? I thought that hadn't been implemented yet. |
I don't think so based on this comment from yesterday stryker-mutator/stryker-net#3094 (comment)
Maybe it's not a problem as per the comment but you know better of course, |
Member
This should have been fixed by #3000, so the changes related to strong naming should be able to be removed. |
Member
|
I'm moving this back to draft as stryker is very broken, but doesn't error so the green CI legs are lies. |
slang25
force-pushed
the
xunitv3
branch
2 times, most recently
from
b6fbc70 to
5d79e37
Compare
- Replace xunit 2.9.3 and xunit.runner.visualstudio with xunit.v3.mtp-v2 3.2.2 - Update FsCheck.Xunit to FsCheck.Xunit.v3 - Add Microsoft.Testing.Platform 2.1.0 and enable MTP in global.json - Add OutputType Exe to test projects (required by xUnit v3) - Configure Stryker with test-runner mtp for coverage support - Run test executables directly via dotnet exec, bypassing dotnet test --project MSBuild discovery bug (dotnet/sdk#49063) - Update ITestOutputHelper implementations for new xUnit v3 interface - Fix TheoryData type inference with explicit casts - Use TestContext.Current.CancellationToken for test cancellation - Remove obsolete Xunit.Abstractions using directives Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Patch Stryker's MTP test runner to fix three bugs that caused ~55% of mutations to incorrectly survive: 1. Only "failed" execution state was counted as test failure, not "error" 2. EveryTest() sentinel was lost during test result accumulation when the MTP server crashed (GetIdentifiers() returns empty for sentinel) 3. Static field initializer mutations were never killed because the MTP process reuses the same process across mutations, and static fields initialize only once Also add eng/AssemblyVersion.cs to preserve the assembly version in Stryker-compiled assemblies (Stryker skips auto-generated files which caused version mismatch: 1.0.0.0 vs expected 8.0.0.0). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues with the PatchStryker Cake task: 1. Patch file didn't match actual v4.14.1 source (regenerated) 2. Path resolution used wrong working directory (use script-relative path) 3. Build ran from wrong working directory (NuGet packages not found) Also change target-framework from net10.0 to net8.0 since source projects don't target net10.0, causing Stryker framework mismatch on CI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The AssemblyVersion.cs file was unconditionally included via Library.targets, causing build failures in VB.NET and F# projects that import it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The MTP test runner has higher per-session overhead than vstest due to JSON-RPC server process startup per mutation. The legacy mutation job (889 mutants, 2113 tests) needs ~80-90 minutes with MTP vs ~6 minutes with vstest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 0e8450d.
The legacy Polly.csproj targets net6.0 (no net8.0), causing a framework mismatch when Stryker's MTP runner tried to test mutated net6.0 assemblies in a net8.0 test host. This caused mutation testing to deadlock with zero progress output. Removing target-framework lets Stryker auto-detect the correct framework per project, matching the upstream main branch config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The legacy Polly.csproj targets net6.0 (no net8.0) while Polly.Specs targets net8.0+. This TFM mismatch causes Stryker's MTP runner to deadlock during mutation testing — the MTP server process hangs without sending completion signals, blocking all runners indefinitely. Use the default (vstest) test runner for the legacy project only. The test project still uses xUnit v3 with MTP natively; only Stryker's communication protocol changes for this one project. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t-framework" This reverts commit d7868f0.
The MTP runner works locally with concurrency 4 but deadlocks on CI (ubuntu-latest, 2 CPUs). Adding --verbosity debug --log-to-file to capture detailed Stryker logs for the legacy project. The logs will be included in the mutation-report artifact for analysis. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The legacy mutation run (889 mutants × 2113 tests) takes ~79 minutes, exceeding the 60-minute timeout. Stryker's dots reporter is buffered on CI so no output appears until the run completes, making timeouts look like deadlocks. Also removes debug logging added for investigation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR migrates the test infrastructure from xUnit v2 to xUnit v3 with Microsoft Testing Platform (MTP) support.
Changes
Package Updates:
xunit(2.9.3) andxunit.runner.visualstudio(3.1.5) withxunit.v3.mtp-v2(3.2.2)FsCheck.XunittoFsCheck.Xunit.v3Microsoft.Testing.Platform(2.1.0) explicit version pindotnet-strykeralready at 4.14.0 which includes MTP coverage supportBuild Configuration:
<OutputType>Exe</OutputType>to all test projects (required by xUnit v3)global.json"test-runner": "mtp")dotnet execinstead ofdotnet test, working around known SDK bug withdotnet test --projectin MTP mode (Validate historical hacks for test projects that may be harmful for MTP projects dotnet/sdk#49063)--report-spekt-junitfor JUnit XML reporting (MTP-compatible flag from JunitXml.TestLogger)Code Changes:
ITestOutputHelperimplementations to include newOutputproperty andWritemethodsTheoryDatatype inference issues with explicit castsTestContext.Current.CancellationTokenfor test cancellationXunit.Abstractionsusing directivesNotes
dotnet test --projecthas a known MSBuild property discovery bug on .NET 10 SDK with MTP enabled (Validate historical hacks for test projects that may be harmful for MTP projects dotnet/sdk#49063), so we run test executables directly viadotnet execinsteaddotnet testapproach is not viable either