Skip to content

Commit

Permalink
docs(benchmark): update MSM benchmarks with new flags
Browse files Browse the repository at this point in the history
  • Loading branch information
@ashjeong
ashjeong committed Oct 23, 2024
1 parent 8fbe854 commit a7d1d5f
Show file tree
Hide file tree
Showing 7 changed files with 57 additions and 54 deletions.
111 changes: 57 additions & 54 deletions benchmark/msm/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

```
Run on 13th Gen Intel(R) Core(TM) i9-13900K (32 X 5500 MHz CPU s)
Compiler: clang-15
CPU Caches:
L1 Data 48 KiB (x16)
L1 Instruction 32 KiB (x16)
Expand All @@ -17,75 +18,77 @@ CPU Caches:
L2 Unified 4096 KiB (x12)
```

Note: Run with `build --@rules_rust//:extra_rustc_flags="-Ctarget-cpu=native"` in your .bazelrc.user

### Uniform points

```shell
bazel run --config opt --//:has_matplotlib //benchmark/msm:msm_benchmark -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --vendor arkworks --vendor bellman --vendor halo2 --check_results
GOMP_SPINCOUNT=0 bazel run --config maxopt --//:has_matplotlib //benchmark/msm:msm_benchmark -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --vendor arkworks --vendor bellman --vendor halo2 --check_results
```

#### On Intel i9-13900K

| Exponent | Tachyon | Arkworks | Bellman | Halo2 |
| :------: | ------------ | -------- | -------- | -------- |
| 16 | **0.037107** | 0.043005 | 0.079576 | 0.04854 |
| 17 | **0.051857** | 0.078176 | 0.11077 | 0.087005 |
| 18 | **0.10235** | 0.15416 | 0.179148 | 0.146375 |
| 19 | **0.185314** | 0.301129 | 0.360499 | 0.273742 |
| 20 | **0.352276** | 0.592931 | 0.516447 | 0.533842 |
| 21 | **0.630413** | 1.07979 | 1.00715 | 1.01114 |
| 22 | **1.22393** | 2.18742 | 2.00827 | 1.99418 |
| 23 | **2.34737** | 4.34471 | 3.2081 | 3.62107 |
| 16 | **0.028461** | 0.037741 | 0.077416 | 0.045742 |
| 17 | **0.059648** | 0.074936 | 0.105104 | 0.08211 |
| 18 | **0.08743** | 0.12735 | 0.196602 | 0.151715 |
| 19 | **0.181646** | 0.252424 | 0.319185 | 0.282056 |
| 20 | **0.303829** | 0.454595 | 0.471094 | 0.526231 |
| 21 | **0.549287** | 0.951397 | 0.886244 | 1.00624 |
| 22 | **1.11021** | 2.00783 | 1.72011 | 1.9662 |
| 23 | **2.06762** | 3.78478 | 2.76673 | 3.68139 |

![image](/benchmark/msm/msm_benchmark_uniform_ubuntu_i9.png)

#### On Mac M3 Pro

| Exponent | Tachyon | Arkworks | Bellman | Halo2 |
| :------: | ------------ | -------- | -------- | -------- |
| 16 | **0.049285** | 0.052368 | 0.118564 | 0.079168 |
| 17 | **0.088628** | 0.105875 | 0.16704 | 0.142758 |
| 18 | **0.157609** | 0.19186 | 0.299953 | 0.278592 |
| 19 | **0.282686** | 0.351326 | 0.578682 | 0.506371 |
| 20 | **0.571241** | 0.702036 | 0.901252 | 0.974515 |
| 21 | **1.106550** | 1.54553 | 1.63521 | 1.85615 |
| 22 | **2.276600** | 3.35888 | 3.274 | 3.68391 |
| 23 | **4.191330** | 6.41272 | 5.86292 | 6.89936 |
| 16 | **0.046099** | 0.051773 | 0.110882 | 0.09505 |
| 17 | **0.079298** | 0.097698 | 0.166183 | 0.174984 |
| 18 | **0.151962** | 0.173607 | 0.296879 | 0.337657 |
| 19 | **0.287848** | 0.34129 | 0.5563 | 0.592885 |
| 20 | **0.504987** | 0.630489 | 0.840907 | 1.07097 |
| 21 | **0.980302** | 1.33391 | 1.56196 | 1.98335 |
| 22 | **1.89977** | 2.86768 | 3.04392 | 3.9341 |
| 23 | **3.73015** | 5.71419 | 5.45636 | 7.51033 |

![image](/benchmark/msm/msm_benchmark_uniform_mac_m3.png)

### Non-uniform points

```shell
bazel run --config opt --//:has_matplotlib //benchmark/msm:msm_benchmark -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --vendor arkworks --vendor bellman --vendor halo2 --test_set non_uniform --check_results
GOMP_SPINCOUNT=0 bazel run --config maxopt --//:has_matplotlib //benchmark/msm:msm_benchmark -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --vendor arkworks --vendor bellman --vendor halo2 --test_set non_uniform --check_results
```

#### On Intel i9-13900K

| Exponent | Tachyon | Arkworks | Bellman | Halo2 |
| :------: | ------------ | ------------ | -------- | -------- |
| 16 | 0.041756 | **0.038514** | 0.050397 | 0.040222 |
| 17 | **0.06304** | 0.069963 | 0.067314 | 0.07276 |
| 18 | **0.09546** | 0.137724 | 0.126665 | 0.138729 |
| 19 | **0.189892** | 0.261193 | 0.262439 | 0.26166 |
| 20 | **0.313575** | 0.475754 | 0.387448 | 0.51257 |
| 21 | **0.550828** | 0.901562 | 0.666334 | 0.955411 |
| 22 | **1.14408** | 1.61699 | 1.29853 | 1.88522 |
| 23 | **1.91659** | 3.13911 | 2.16368 | 3.46701 |
| Exponent | Tachyon | Arkworks | Bellman | Halo2 |
| :------: | ------------ | -------- | -------- | -------- |
| 16 | **0.030188** | 0.033608 | 0.057795 | 0.060642 |
| 17 | **0.048851** | 0.064059 | 0.132584 | 0.099568 |
| 18 | **0.080146** | 0.121525 | 0.124192 | 0.147496 |
| 19 | **0.147626** | 0.227517 | 0.234429 | 0.27793 |
| 20 | **0.289661** | 0.445139 | 0.341189 | 0.509375 |
| 21 | **0.495707** | 0.801975 | 0.702259 | 1.0386 |
| 22 | **0.993738** | 1.51266 | 1.24812 | 1.88462 |
| 23 | **1.69944** | 3.07904 | 2.00071 | 3.57452 |

![image](/benchmark/msm/msm_benchmark_non_uniform_ubuntu_i9.png)

#### On Mac M3 Pro

| Exponent | Tachyon | Arkworks | Bellman | Halo2 |
| :------: | ------------ | -------- | -------- | -------- |
| 16 | **0.040045** | 0.049862 | 0.083292 | 0.082423 |
| 17 | **0.074318** | 0.091205 | 0.128679 | 0.157111 |
| 18 | **0.140125** | 0.177842 | 0.232212 | 0.310648 |
| 19 | **0.287691** | 0.330268 | 0.437016 | 0.535915 |
| 20 | **0.55437** | 0.651841 | 0.713282 | 1.01238 |
| 21 | **1.01053** | 1.36348 | 1.29945 | 1.75816 |
| 22 | **2.00677** | 2.56 | 2.49532 | 3.55769 |
| 23 | **4.02119** | 5.2982 | 4.56454 | 7.11582 |
| 16 | **0.040954** | 0.046663 | 0.076068 | 0.089352 |
| 17 | **0.069956** | 0.089339 | 0.119363 | 0.166812 |
| 18 | **0.146869** | 0.163578 | 0.225768 | 0.326553 |
| 19 | **0.268475** | 0.302439 | 0.460063 | 0.579915 |
| 20 | **0.501956** | 0.627272 | 0.723071 | 1.09316 |
| 21 | **0.920728** | 1.20662 | 1.22352 | 1.98457 |
| 22 | **1.78902** | 2.40124 | 2.24543 | 3.83765 |
| 23 | **3.47906** | 4.70651 | 4.13381 | 7.43978 |

![image](/benchmark/msm/msm_benchmark_non_uniform_mac_m3.png)

Expand All @@ -94,41 +97,41 @@ bazel run --config opt --//:has_matplotlib //benchmark/msm:msm_benchmark -- -k 1
### Uniform points

```shell
bazel run --config opt --//:has_matplotlib --config cuda //benchmark/msm:msm_benchmark_gpu -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --test_set non_uniform --check_results
GOMP_SPINCOUNT=0 bazel run --config maxopt --//:has_matplotlib --config cuda //benchmark/msm:msm_benchmark_gpu -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --test_set non_uniform --check_results
```

#### On RTX-4090

| Exponent | Tachyon CPU | Tachyon GPU |
| :------: | ----------- | ------------ |
| 16 | 0.044828 | **0.02644** |
| 17 | 0.070154 | **0.008896** |
| 18 | 0.120668 | **0.015391** |
| 19 | 0.191098 | **0.021299** |
| 20 | 0.369561 | **0.032302** |
| 21 | 0.675571 | **0.060279** |
| 22 | 1.26675 | **0.166554** |
| 23 | 2.41323 | **0.219751** |
| 16 | 0.026688 | **0.01981** |
| 17 | 0.041291 | **0.006624** |
| 18 | 0.081467 | **0.008306** |
| 19 | 0.148929 | **0.012553** |
| 20 | 0.260831 | **0.02423** |
| 21 | 0.474542 | **0.044591** |
| 22 | 0.921276 | **0.088349** |
| 23 | 1.70264 | **0.162646** |

![image](/benchmark/msm/msm_benchmark_uniform_ubuntu_rtx_4090.png)

### Non-uniform points

```shell
bazel run --config opt --//:has_matplotlib --config cuda //benchmark/msm:msm_benchmark_gpu -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --check_results
GOMP_SPINCOUNT=0 bazel run --config maxopt --//:has_matplotlib --config cuda //benchmark/msm:msm_benchmark_gpu -- -k 16 -k 17 -k 18 -k 19 -k 20 -k 21 -k 22 -k 23 --check_results
```

#### On RTX-4090

| Exponent | Tachyon CPU | Tachyon GPU |
| :------: | ----------- | ------------ |
| 16 | 0.044986 | **0.025648** |
| 17 | 0.061027 | **0.010882** |
| 18 | 0.108204 | **0.012794** |
| 19 | 0.182035 | **0.022456** |
| 20 | 0.326646 | **0.033684** |
| 21 | 0.589097 | **0.060135** |
| 22 | 1.11816 | **0.169485** |
| 23 | 2.11566 | **0.219624** |
| 16 | 0.029045 | **0.020228** |
| 17 | 0.047588 | **0.006565** |
| 18 | 0.089673 | **0.008864** |
| 19 | 0.164875 | **0.012308** |
| 20 | 0.29135 | **0.023396** |
| 21 | 0.541067 | **0.043512** |
| 22 | 1.0379 | **0.08407** |
| 23 | 2.08601 | **0.157046** |

![image](/benchmark/msm/msm_benchmark_non_uniform_ubuntu_rtx_4090.png)
Binary file modified benchmark/msm/msm_benchmark_non_uniform_mac_m3.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/msm/msm_benchmark_non_uniform_ubuntu_i9.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/msm/msm_benchmark_non_uniform_ubuntu_rtx_4090.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/msm/msm_benchmark_uniform_mac_m3.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/msm/msm_benchmark_uniform_ubuntu_i9.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified benchmark/msm/msm_benchmark_uniform_ubuntu_rtx_4090.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit a7d1d5f

Please sign in to comment.