diff --git a/README.md b/README.md index 5838e8e7a8..ffb9a22e69 100644 --- a/README.md +++ b/README.md @@ -222,21 +222,17 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0] Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | Median | -|------------- |--------- |--------- |---------:|---------:|----------:|---------:| -| Build_TUnit | .NET 8.0 | .NET 8.0 | 897.3 ms | 17.82 ms | 24.40 ms | 895.7 ms | -| Build_NUnit | .NET 8.0 | .NET 8.0 | 811.7 ms | 14.97 ms | 13.27 ms | 811.5 ms | -| Build_xUnit | .NET 8.0 | .NET 8.0 | 810.7 ms | 15.23 ms | 12.72 ms | 810.0 ms | -| Build_MSTest | .NET 8.0 | .NET 8.0 | 837.0 ms | 12.77 ms | 11.32 ms | 838.5 ms | -| Build_TUnit | .NET 9.0 | .NET 9.0 | 886.8 ms | 17.55 ms | 30.27 ms | 873.2 ms | -| Build_NUnit | .NET 9.0 | .NET 9.0 | 864.5 ms | 17.22 ms | 48.57 ms | 850.5 ms | -| Build_xUnit | .NET 9.0 | .NET 9.0 | 961.9 ms | 55.31 ms | 155.11 ms | 919.2 ms | -| Build_MSTest | .NET 9.0 | .NET 9.0 | 875.1 ms | 17.45 ms | 27.16 ms | 873.4 ms | +| Method | Mean | Error | StdDev | Median | +|------------- |---------:|---------:|----------:|---------:| +| Build_TUnit | 903.7 ms | 17.34 ms | 46.89 ms | 893.1 ms | +| Build_NUnit | 822.3 ms | 16.40 ms | 32.38 ms | 809.7 ms | +| Build_xUnit | 810.2 ms | 11.78 ms | 10.45 ms | 811.4 ms | +| Build_MSTest | 994.3 ms | 62.47 ms | 181.23 ms | 906.0 ms | @@ -248,21 +244,17 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|------------- |--------- |--------- |--------:|---------:|---------:| -| Build_TUnit | .NET 8.0 | .NET 8.0 | 1.743 s | 0.0341 s | 0.0393 s | -| Build_NUnit | .NET 8.0 | .NET 8.0 | 1.564 s | 0.0291 s | 0.0258 s | -| Build_xUnit | .NET 8.0 | .NET 8.0 | 1.579 s | 0.0304 s | 0.0285 s | -| Build_MSTest | .NET 8.0 | .NET 8.0 | 1.640 s | 0.0296 s | 0.0262 s | -| Build_TUnit | .NET 9.0 | .NET 9.0 | 1.730 s | 0.0296 s | 0.0262 s | -| Build_NUnit | .NET 9.0 | .NET 9.0 | 1.575 s | 0.0302 s | 0.0252 s | -| Build_xUnit | .NET 9.0 | .NET 9.0 | 1.536 s | 0.0270 s | 0.0341 s | -| Build_MSTest | .NET 9.0 | .NET 9.0 | 1.599 s | 0.0171 s | 0.0133 s | +| Method | Mean | Error | StdDev | +|------------- |--------:|---------:|---------:| +| Build_TUnit | 1.821 s | 0.0352 s | 0.0376 s | +| Build_NUnit | 1.623 s | 0.0236 s | 0.0221 s | +| Build_xUnit | 1.651 s | 0.0272 s | 0.0255 s | +| Build_MSTest | 1.773 s | 0.0345 s | 0.0354 s | @@ -274,21 +266,17 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|------------- |--------- |--------- |--------:|---------:|---------:| -| Build_TUnit | .NET 8.0 | .NET 8.0 | 1.682 s | 0.0231 s | 0.0205 s | -| Build_NUnit | .NET 8.0 | .NET 8.0 | 1.515 s | 0.0274 s | 0.0257 s | -| Build_xUnit | .NET 8.0 | .NET 8.0 | 1.523 s | 0.0178 s | 0.0158 s | -| Build_MSTest | .NET 8.0 | .NET 8.0 | 1.582 s | 0.0188 s | 0.0176 s | -| Build_TUnit | .NET 9.0 | .NET 9.0 | 1.699 s | 0.0323 s | 0.0302 s | -| Build_NUnit | .NET 9.0 | .NET 9.0 | 1.552 s | 0.0255 s | 0.0238 s | -| Build_xUnit | .NET 9.0 | .NET 9.0 | 1.560 s | 0.0192 s | 0.0180 s | -| Build_MSTest | .NET 9.0 | .NET 9.0 | 1.596 s | 0.0197 s | 0.0175 s | +| Method | Mean | Error | StdDev | +|------------- |--------:|---------:|---------:| +| Build_TUnit | 1.697 s | 0.0298 s | 0.0279 s | +| Build_NUnit | 1.551 s | 0.0292 s | 0.0273 s | +| Build_xUnit | 1.570 s | 0.0195 s | 0.0183 s | +| Build_MSTest | 1.591 s | 0.0237 s | 0.0222 s | ### Scenario: A single test that completes instantly (including spawning a new process and initialising the test framework) @@ -301,23 +289,18 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0] Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | Median | -|---------- |--------- |--------- |---------:|---------:|----------:|---------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 176.8 ms | 11.49 ms | 33.70 ms | 167.5 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 470.3 ms | 8.29 ms | 19.85 ms | 464.9 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 694.4 ms | 4.17 ms | 3.90 ms | 693.7 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 803.3 ms | 47.01 ms | 136.39 ms | 742.2 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 663.2 ms | 13.17 ms | 27.49 ms | 656.9 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 117.5 ms | 0.25 ms | 0.21 ms | 117.4 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 461.0 ms | 8.99 ms | 11.37 ms | 457.2 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 701.1 ms | 12.89 ms | 11.42 ms | 697.3 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 683.3 ms | 8.77 ms | 7.33 ms | 683.5 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 626.4 ms | 12.22 ms | 11.43 ms | 625.3 ms | +| Method | Mean | Error | StdDev | +|---------- |-----------:|---------:|----------:| +| TUnit_AOT | 173.9 ms | 7.63 ms | 22.00 ms | +| TUnit | 790.1 ms | 24.94 ms | 72.36 ms | +| NUnit | 1,156.1 ms | 33.52 ms | 98.84 ms | +| xUnit | 1,244.1 ms | 39.40 ms | 115.56 ms | +| MSTest | 1,068.2 ms | 43.01 ms | 125.47 ms | @@ -329,23 +312,18 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|---------- |--------- |--------- |------------:|----------:|----------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 87.16 ms | 1.720 ms | 4.219 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 849.04 ms | 16.466 ms | 26.589 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 1,334.09 ms | 15.615 ms | 13.040 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 1,328.56 ms | 18.563 ms | 16.455 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 1,188.48 ms | 16.984 ms | 15.887 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 64.06 ms | 0.909 ms | 0.806 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 821.84 ms | 16.381 ms | 27.369 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 1,319.33 ms | 18.861 ms | 17.643 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 1,287.32 ms | 7.951 ms | 7.048 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 1,165.25 ms | 12.304 ms | 11.509 ms | +| Method | Mean | Error | StdDev | +|---------- |------------:|----------:|----------:| +| TUnit_AOT | 63.44 ms | 0.631 ms | 0.559 ms | +| TUnit | 823.27 ms | 15.916 ms | 30.665 ms | +| NUnit | 1,306.20 ms | 9.470 ms | 8.395 ms | +| xUnit | 1,294.08 ms | 12.275 ms | 9.584 ms | +| MSTest | 1,174.94 ms | 7.846 ms | 7.339 ms | @@ -357,23 +335,18 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | Median | -|---------- |--------- |--------- |-----------:|---------:|---------:|-----------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 124.3 ms | 2.37 ms | 1.85 ms | 124.9 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 850.2 ms | 16.94 ms | 24.29 ms | 851.1 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 1,315.3 ms | 10.02 ms | 9.38 ms | 1,317.7 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 1,304.1 ms | 10.61 ms | 9.92 ms | 1,303.9 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 1,186.5 ms | 17.64 ms | 16.50 ms | 1,184.7 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 125.6 ms | 1.37 ms | 1.21 ms | 124.9 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 867.0 ms | 17.28 ms | 25.33 ms | 851.3 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 1,339.3 ms | 13.95 ms | 12.37 ms | 1,340.3 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 1,317.5 ms | 9.31 ms | 8.70 ms | 1,319.0 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 1,194.8 ms | 14.55 ms | 13.61 ms | 1,198.2 ms | +| Method | Mean | Error | StdDev | +|---------- |-----------:|---------:|---------:| +| TUnit_AOT | 126.2 ms | 2.52 ms | 5.75 ms | +| TUnit | 924.5 ms | 18.45 ms | 29.80 ms | +| NUnit | 1,403.8 ms | 21.51 ms | 20.12 ms | +| xUnit | 1,392.8 ms | 27.59 ms | 25.80 ms | +| MSTest | 1,255.5 ms | 19.30 ms | 18.05 ms | ### Scenario: A test that takes 50ms to execute, repeated 100 times (including spawning a new process and initialising the test framework) @@ -386,23 +359,18 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0] Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|---------- |--------- |--------- |------------:|----------:|----------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 287.5 ms | 15.18 ms | 44.53 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 630.7 ms | 20.78 ms | 60.27 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 14,472.7 ms | 280.12 ms | 676.53 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 14,554.4 ms | 290.54 ms | 538.54 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 14,708.5 ms | 280.04 ms | 572.05 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 291.2 ms | 15.71 ms | 46.31 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 619.3 ms | 20.14 ms | 59.37 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 14,166.0 ms | 275.39 ms | 394.95 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 14,356.9 ms | 286.78 ms | 494.68 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 14,423.7 ms | 283.26 ms | 532.04 ms | +| Method | Mean | Error | StdDev | +|---------- |------------:|----------:|----------:| +| TUnit_AOT | 318.9 ms | 22.53 ms | 66.43 ms | +| TUnit | 875.6 ms | 28.68 ms | 83.21 ms | +| NUnit | 14,388.3 ms | 280.40 ms | 627.16 ms | +| xUnit | 14,741.0 ms | 294.59 ms | 621.39 ms | +| MSTest | 14,329.5 ms | 284.27 ms | 490.35 ms | @@ -414,23 +382,18 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|---------- |--------- |--------- |-----------:|---------:|---------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 134.9 ms | 2.69 ms | 6.07 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 918.3 ms | 18.17 ms | 30.85 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 6,556.3 ms | 21.49 ms | 19.05 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 6,530.6 ms | 26.33 ms | 24.63 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 6,471.2 ms | 10.65 ms | 9.97 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 124.2 ms | 2.48 ms | 6.07 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 892.6 ms | 17.64 ms | 29.95 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 6,499.6 ms | 23.38 ms | 21.87 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 6,467.8 ms | 18.79 ms | 16.66 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 6,442.9 ms | 28.44 ms | 25.21 ms | +| Method | Mean | Error | StdDev | +|---------- |-----------:|---------:|---------:| +| TUnit_AOT | 126.8 ms | 2.52 ms | 5.20 ms | +| TUnit | 912.4 ms | 18.23 ms | 21.00 ms | +| NUnit | 6,515.9 ms | 35.58 ms | 33.28 ms | +| xUnit | 6,459.6 ms | 22.21 ms | 19.69 ms | +| MSTest | 6,450.3 ms | 22.54 ms | 19.98 ms | @@ -442,23 +405,18 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V) AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores .NET SDK 9.0.100-rc.2.24474.11 [Host] : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 - .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2 .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2 +Job=.NET 9.0 Runtime=.NET 9.0 ``` -| Method | Job | Runtime | Mean | Error | StdDev | -|---------- |--------- |--------- |-----------:|---------:|---------:| -| TUnit_AOT | .NET 8.0 | .NET 8.0 | 186.4 ms | 3.43 ms | 2.87 ms | -| TUnit | .NET 8.0 | .NET 8.0 | 924.8 ms | 18.33 ms | 23.83 ms | -| NUnit | .NET 8.0 | .NET 8.0 | 7,565.0 ms | 17.16 ms | 15.21 ms | -| xUnit | .NET 8.0 | .NET 8.0 | 7,549.5 ms | 19.40 ms | 17.20 ms | -| MSTest | .NET 8.0 | .NET 8.0 | 7,511.5 ms | 28.97 ms | 27.10 ms | -| TUnit_AOT | .NET 9.0 | .NET 9.0 | 178.2 ms | 3.54 ms | 8.33 ms | -| TUnit | .NET 9.0 | .NET 9.0 | 968.3 ms | 19.14 ms | 26.83 ms | -| NUnit | .NET 9.0 | .NET 9.0 | 7,575.6 ms | 23.91 ms | 21.20 ms | -| xUnit | .NET 9.0 | .NET 9.0 | 7,548.2 ms | 21.35 ms | 19.97 ms | -| MSTest | .NET 9.0 | .NET 9.0 | 7,485.9 ms | 29.34 ms | 26.01 ms | +| Method | Mean | Error | StdDev | +|---------- |-----------:|---------:|---------:| +| TUnit_AOT | 178.6 ms | 3.56 ms | 9.20 ms | +| TUnit | 962.4 ms | 18.89 ms | 25.86 ms | +| NUnit | 7,583.5 ms | 21.48 ms | 20.09 ms | +| xUnit | 7,565.6 ms | 41.29 ms | 38.62 ms | +| MSTest | 7,494.7 ms | 16.20 ms | 15.15 ms |