Merge remote-tracking branch 'origin/main'

thomhurst · Oct 12, 2024 · 909ed11 · 909ed11
2 parents e7521c0 + 9b36923
commit 909ed11
Showing 1 changed file with 69 additions and 111 deletions.
diff --git a/README.md b/README.md
@@ -222,21 +222,17 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0]
 Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method       | Job      | Runtime  | Mean     | Error    | StdDev    | Median   |
-|------------- |--------- |--------- |---------:|---------:|----------:|---------:|
-| Build_TUnit  | .NET 8.0 | .NET 8.0 | 897.3 ms | 17.82 ms |  24.40 ms | 895.7 ms |
-| Build_NUnit  | .NET 8.0 | .NET 8.0 | 811.7 ms | 14.97 ms |  13.27 ms | 811.5 ms |
-| Build_xUnit  | .NET 8.0 | .NET 8.0 | 810.7 ms | 15.23 ms |  12.72 ms | 810.0 ms |
-| Build_MSTest | .NET 8.0 | .NET 8.0 | 837.0 ms | 12.77 ms |  11.32 ms | 838.5 ms |
-| Build_TUnit  | .NET 9.0 | .NET 9.0 | 886.8 ms | 17.55 ms |  30.27 ms | 873.2 ms |
-| Build_NUnit  | .NET 9.0 | .NET 9.0 | 864.5 ms | 17.22 ms |  48.57 ms | 850.5 ms |
-| Build_xUnit  | .NET 9.0 | .NET 9.0 | 961.9 ms | 55.31 ms | 155.11 ms | 919.2 ms |
-| Build_MSTest | .NET 9.0 | .NET 9.0 | 875.1 ms | 17.45 ms |  27.16 ms | 873.4 ms |
+| Method       | Mean     | Error    | StdDev    | Median   |
+|------------- |---------:|---------:|----------:|---------:|
+| Build_TUnit  | 903.7 ms | 17.34 ms |  46.89 ms | 893.1 ms |
+| Build_NUnit  | 822.3 ms | 16.40 ms |  32.38 ms | 809.7 ms |
+| Build_xUnit  | 810.2 ms | 11.78 ms |  10.45 ms | 811.4 ms |
+| Build_MSTest | 994.3 ms | 62.47 ms | 181.23 ms | 906.0 ms |
 
 
 
@@ -248,21 +244,17 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method       | Job      | Runtime  | Mean    | Error    | StdDev   |
-|------------- |--------- |--------- |--------:|---------:|---------:|
-| Build_TUnit  | .NET 8.0 | .NET 8.0 | 1.743 s | 0.0341 s | 0.0393 s |
-| Build_NUnit  | .NET 8.0 | .NET 8.0 | 1.564 s | 0.0291 s | 0.0258 s |
-| Build_xUnit  | .NET 8.0 | .NET 8.0 | 1.579 s | 0.0304 s | 0.0285 s |
-| Build_MSTest | .NET 8.0 | .NET 8.0 | 1.640 s | 0.0296 s | 0.0262 s |
-| Build_TUnit  | .NET 9.0 | .NET 9.0 | 1.730 s | 0.0296 s | 0.0262 s |
-| Build_NUnit  | .NET 9.0 | .NET 9.0 | 1.575 s | 0.0302 s | 0.0252 s |
-| Build_xUnit  | .NET 9.0 | .NET 9.0 | 1.536 s | 0.0270 s | 0.0341 s |
-| Build_MSTest | .NET 9.0 | .NET 9.0 | 1.599 s | 0.0171 s | 0.0133 s |
+| Method       | Mean    | Error    | StdDev   |
+|------------- |--------:|---------:|---------:|
+| Build_TUnit  | 1.821 s | 0.0352 s | 0.0376 s |
+| Build_NUnit  | 1.623 s | 0.0236 s | 0.0221 s |
+| Build_xUnit  | 1.651 s | 0.0272 s | 0.0255 s |
+| Build_MSTest | 1.773 s | 0.0345 s | 0.0354 s |
 
 
 
@@ -274,21 +266,17 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method       | Job      | Runtime  | Mean    | Error    | StdDev   |
-|------------- |--------- |--------- |--------:|---------:|---------:|
-| Build_TUnit  | .NET 8.0 | .NET 8.0 | 1.682 s | 0.0231 s | 0.0205 s |
-| Build_NUnit  | .NET 8.0 | .NET 8.0 | 1.515 s | 0.0274 s | 0.0257 s |
-| Build_xUnit  | .NET 8.0 | .NET 8.0 | 1.523 s | 0.0178 s | 0.0158 s |
-| Build_MSTest | .NET 8.0 | .NET 8.0 | 1.582 s | 0.0188 s | 0.0176 s |
-| Build_TUnit  | .NET 9.0 | .NET 9.0 | 1.699 s | 0.0323 s | 0.0302 s |
-| Build_NUnit  | .NET 9.0 | .NET 9.0 | 1.552 s | 0.0255 s | 0.0238 s |
-| Build_xUnit  | .NET 9.0 | .NET 9.0 | 1.560 s | 0.0192 s | 0.0180 s |
-| Build_MSTest | .NET 9.0 | .NET 9.0 | 1.596 s | 0.0197 s | 0.0175 s |
+| Method       | Mean    | Error    | StdDev   |
+|------------- |--------:|---------:|---------:|
+| Build_TUnit  | 1.697 s | 0.0298 s | 0.0279 s |
+| Build_NUnit  | 1.551 s | 0.0292 s | 0.0273 s |
+| Build_xUnit  | 1.570 s | 0.0195 s | 0.0183 s |
+| Build_MSTest | 1.591 s | 0.0237 s | 0.0222 s |
 
 
 ### Scenario: A single test that completes instantly (including spawning a new process and initialising the test framework)
@@ -301,23 +289,18 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0]
 Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean     | Error    | StdDev    | Median   |
-|---------- |--------- |--------- |---------:|---------:|----------:|---------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 | 176.8 ms | 11.49 ms |  33.70 ms | 167.5 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 | 470.3 ms |  8.29 ms |  19.85 ms | 464.9 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 694.4 ms |  4.17 ms |   3.90 ms | 693.7 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 803.3 ms | 47.01 ms | 136.39 ms | 742.2 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 663.2 ms | 13.17 ms |  27.49 ms | 656.9 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 | 117.5 ms |  0.25 ms |   0.21 ms | 117.4 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 | 461.0 ms |  8.99 ms |  11.37 ms | 457.2 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 701.1 ms | 12.89 ms |  11.42 ms | 697.3 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 683.3 ms |  8.77 ms |   7.33 ms | 683.5 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 626.4 ms | 12.22 ms |  11.43 ms | 625.3 ms |
+| Method    | Mean       | Error    | StdDev    |
+|---------- |-----------:|---------:|----------:|
+| TUnit_AOT |   173.9 ms |  7.63 ms |  22.00 ms |
+| TUnit     |   790.1 ms | 24.94 ms |  72.36 ms |
+| NUnit     | 1,156.1 ms | 33.52 ms |  98.84 ms |
+| xUnit     | 1,244.1 ms | 39.40 ms | 115.56 ms |
+| MSTest    | 1,068.2 ms | 43.01 ms | 125.47 ms |
 
 
 
@@ -329,23 +312,18 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean        | Error     | StdDev    |
-|---------- |--------- |--------- |------------:|----------:|----------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 |    87.16 ms |  1.720 ms |  4.219 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 |   849.04 ms | 16.466 ms | 26.589 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 1,334.09 ms | 15.615 ms | 13.040 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 1,328.56 ms | 18.563 ms | 16.455 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 1,188.48 ms | 16.984 ms | 15.887 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 |    64.06 ms |  0.909 ms |  0.806 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 |   821.84 ms | 16.381 ms | 27.369 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 1,319.33 ms | 18.861 ms | 17.643 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 1,287.32 ms |  7.951 ms |  7.048 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 1,165.25 ms | 12.304 ms | 11.509 ms |
+| Method    | Mean        | Error     | StdDev    |
+|---------- |------------:|----------:|----------:|
+| TUnit_AOT |    63.44 ms |  0.631 ms |  0.559 ms |
+| TUnit     |   823.27 ms | 15.916 ms | 30.665 ms |
+| NUnit     | 1,306.20 ms |  9.470 ms |  8.395 ms |
+| xUnit     | 1,294.08 ms | 12.275 ms |  9.584 ms |
+| MSTest    | 1,174.94 ms |  7.846 ms |  7.339 ms |
 
 
 
@@ -357,23 +335,18 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean       | Error    | StdDev   | Median     |
-|---------- |--------- |--------- |-----------:|---------:|---------:|-----------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 |   124.3 ms |  2.37 ms |  1.85 ms |   124.9 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 |   850.2 ms | 16.94 ms | 24.29 ms |   851.1 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 1,315.3 ms | 10.02 ms |  9.38 ms | 1,317.7 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 1,304.1 ms | 10.61 ms |  9.92 ms | 1,303.9 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 1,186.5 ms | 17.64 ms | 16.50 ms | 1,184.7 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 |   125.6 ms |  1.37 ms |  1.21 ms |   124.9 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 |   867.0 ms | 17.28 ms | 25.33 ms |   851.3 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 1,339.3 ms | 13.95 ms | 12.37 ms | 1,340.3 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 1,317.5 ms |  9.31 ms |  8.70 ms | 1,319.0 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 1,194.8 ms | 14.55 ms | 13.61 ms | 1,198.2 ms |
+| Method    | Mean       | Error    | StdDev   |
+|---------- |-----------:|---------:|---------:|
+| TUnit_AOT |   126.2 ms |  2.52 ms |  5.75 ms |
+| TUnit     |   924.5 ms | 18.45 ms | 29.80 ms |
+| NUnit     | 1,403.8 ms | 21.51 ms | 20.12 ms |
+| xUnit     | 1,392.8 ms | 27.59 ms | 25.80 ms |
+| MSTest    | 1,255.5 ms | 19.30 ms | 18.05 ms |
 
 
 ### Scenario: A test that takes 50ms to execute, repeated 100 times (including spawning a new process and initialising the test framework)
@@ -386,23 +359,18 @@ BenchmarkDotNet v0.14.0, macOS Sonoma 14.7 (23H124) [Darwin 23.6.0]
 Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), Arm64 RyuJIT AdvSIMD
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), Arm64 RyuJIT AdvSIMD
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean        | Error     | StdDev    |
-|---------- |--------- |--------- |------------:|----------:|----------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 |    287.5 ms |  15.18 ms |  44.53 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 |    630.7 ms |  20.78 ms |  60.27 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 14,472.7 ms | 280.12 ms | 676.53 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 14,554.4 ms | 290.54 ms | 538.54 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 14,708.5 ms | 280.04 ms | 572.05 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 |    291.2 ms |  15.71 ms |  46.31 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 |    619.3 ms |  20.14 ms |  59.37 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 14,166.0 ms | 275.39 ms | 394.95 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 14,356.9 ms | 286.78 ms | 494.68 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 14,423.7 ms | 283.26 ms | 532.04 ms |
+| Method    | Mean        | Error     | StdDev    |
+|---------- |------------:|----------:|----------:|
+| TUnit_AOT |    318.9 ms |  22.53 ms |  66.43 ms |
+| TUnit     |    875.6 ms |  28.68 ms |  83.21 ms |
+| NUnit     | 14,388.3 ms | 280.40 ms | 627.16 ms |
+| xUnit     | 14,741.0 ms | 294.59 ms | 621.39 ms |
+| MSTest    | 14,329.5 ms | 284.27 ms | 490.35 ms |
 
 
 
@@ -414,23 +382,18 @@ BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean       | Error    | StdDev   |
-|---------- |--------- |--------- |-----------:|---------:|---------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 |   134.9 ms |  2.69 ms |  6.07 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 |   918.3 ms | 18.17 ms | 30.85 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 6,556.3 ms | 21.49 ms | 19.05 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 6,530.6 ms | 26.33 ms | 24.63 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 6,471.2 ms | 10.65 ms |  9.97 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 |   124.2 ms |  2.48 ms |  6.07 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 |   892.6 ms | 17.64 ms | 29.95 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 6,499.6 ms | 23.38 ms | 21.87 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 6,467.8 ms | 18.79 ms | 16.66 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 6,442.9 ms | 28.44 ms | 25.21 ms |
+| Method    | Mean       | Error    | StdDev   |
+|---------- |-----------:|---------:|---------:|
+| TUnit_AOT |   126.8 ms |  2.52 ms |  5.20 ms |
+| TUnit     |   912.4 ms | 18.23 ms | 21.00 ms |
+| NUnit     | 6,515.9 ms | 35.58 ms | 33.28 ms |
+| xUnit     | 6,459.6 ms | 22.21 ms | 19.69 ms |
+| MSTest    | 6,450.3 ms | 22.54 ms | 19.98 ms |
 
 
 
@@ -442,23 +405,18 @@ BenchmarkDotNet v0.14.0, Windows 10 (10.0.20348.2700) (Hyper-V)
 AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
 .NET SDK 9.0.100-rc.2.24474.11
   [Host]   : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
-  .NET 8.0 : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
   .NET 9.0 : .NET 9.0.0 (9.0.24.47305), X64 RyuJIT AVX2
 
+Job=.NET 9.0  Runtime=.NET 9.0  
 
 ```
-| Method    | Job      | Runtime  | Mean       | Error    | StdDev   |
-|---------- |--------- |--------- |-----------:|---------:|---------:|
-| TUnit_AOT | .NET 8.0 | .NET 8.0 |   186.4 ms |  3.43 ms |  2.87 ms |
-| TUnit     | .NET 8.0 | .NET 8.0 |   924.8 ms | 18.33 ms | 23.83 ms |
-| NUnit     | .NET 8.0 | .NET 8.0 | 7,565.0 ms | 17.16 ms | 15.21 ms |
-| xUnit     | .NET 8.0 | .NET 8.0 | 7,549.5 ms | 19.40 ms | 17.20 ms |
-| MSTest    | .NET 8.0 | .NET 8.0 | 7,511.5 ms | 28.97 ms | 27.10 ms |
-| TUnit_AOT | .NET 9.0 | .NET 9.0 |   178.2 ms |  3.54 ms |  8.33 ms |
-| TUnit     | .NET 9.0 | .NET 9.0 |   968.3 ms | 19.14 ms | 26.83 ms |
-| NUnit     | .NET 9.0 | .NET 9.0 | 7,575.6 ms | 23.91 ms | 21.20 ms |
-| xUnit     | .NET 9.0 | .NET 9.0 | 7,548.2 ms | 21.35 ms | 19.97 ms |
-| MSTest    | .NET 9.0 | .NET 9.0 | 7,485.9 ms | 29.34 ms | 26.01 ms |
+| Method    | Mean       | Error    | StdDev   |
+|---------- |-----------:|---------:|---------:|
+| TUnit_AOT |   178.6 ms |  3.56 ms |  9.20 ms |
+| TUnit     |   962.4 ms | 18.89 ms | 25.86 ms |
+| NUnit     | 7,583.5 ms | 21.48 ms | 20.09 ms |
+| xUnit     | 7,565.6 ms | 41.29 ms | 38.62 ms |
+| MSTest    | 7,494.7 ms | 16.20 ms | 15.15 ms |