Skip to content

Commit

Permalink
Added parameterized benchmarks, convergence detection
Browse files Browse the repository at this point in the history
  • Loading branch information
@Izaakwltn
Izaakwltn committed Oct 31, 2024
1 parent e4303de commit 4eb3080
Show file tree
Hide file tree
Showing 6 changed files with 450 additions and 143 deletions.
104 changes: 80 additions & 24 deletions benchmarking/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Coalton benchmarking prints to the repl by default.
This setting can be turned off with:

```
(cl:setf *coalton-verbose-benchmarking* cl:nil)
(cell:write! *coalton-verbose-benchmarking* False)
```

### Printing width
Expand All @@ -27,14 +27,14 @@ Coalton benchmarks print to the repl at 90 characters wide by default.
This can be changed using:

```
(cl:setf *coalton-benchmark-width* 90)
(cell:write! *benchmark-width* <UFix>)
```

### Print time in cientific notation
By default, times are printed using scientific notation. This can be turned off using:

```
(cl:setf *coalton-benchmark-sci-notation* cl:nil)
(ell:write! *coalton-benchmark-sci-notation* False)
```

## Defining benchmarks:
Expand All @@ -43,7 +43,7 @@ Benchmarks can be defined in any Coalton package (that imports or nicknames `#:c

```
;; Defining a Coalton benchmark
(define-benchmark stak 1000 ; iterations
(define-benchmark stak 1000 ; samples
(fn ()
(stak 18 12 6)
Unit))
Expand All @@ -56,27 +56,58 @@ Benchmarks can be defined in any Coalton package (that imports or nicknames `#:c
Unit)))
```

Parameterized benchmarks allow for multiple inputs to the same function:

```
(define-parameterized-benchmark rec-fib 1000
(fn (x)
(fib x)
Unit)
(seq:make 10 15 20 25))
```
Using `:detect-convergence?` will add standard deviation information to your run, as well as dynamic sample scaling.

### Dynamic sample scaling


With `:detect-convergence?` selected, benchmarks will run one sample at a time, collecting live standard deviation information for timings:

```
(define-parameterized-benchmark rec-fib-generic 1000
(fn (x)
(fib-generic-wrapped x)
Unit)
(seq:make 10 15 20 25)
:detect-convergence? cl:t)
```

When running this mode, the `samples` input is interpreted as the minimum samples, and the benchmark will continue running until the standard deviation of time converges. This ensures the minimum amount of samples before a reliable average.

This mode is most effective on benchmarks with significant individual sample times.

## Running individual benchmarks

Individual benchmarks can be run with `#'run-benchmark`, as long as the benchmark is defined in the current package.

`#'run-benchmark` returns a `BenchmarkResults` object.

```
COALTON-BENCHMARKS> (coalton (run-benchmark "tak"))
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmark "rec-fib0"))
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Benchmark tak │
│ Benchmark rec-fib0 │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Coalton development mode without heuristic inlining │
├───────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┤
│ Benchmark │ Time Elapsed │ Bytes consed │ # Iterations │
├───────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┤
│ TAK │ 9.4079e-1 s │ 0 │ 1000 │
└───────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
└──────────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
#.(BENCHMARKRESULTS "TAK" 1000 940788 #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB0 1000 2371 2 #.NONE #.(SOME 0))
```

## Running package benchmarks
Expand All @@ -85,24 +116,49 @@ Package benchmarks can be run with `#'run-package-benchmarks`.

`#'run-package-benchmarks` returns a `PackageBenchmarkResults` object.

Local packages can be run with `#'run-benchmarks`.

```
COALTON-BENCHMARKS> (coalton (run-package-benchmarks "coalton-benchmarks/gabriel/tak"))
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmarks))
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Package 'coalton-benchmarks/gabriel/tak' │
│ Package 'COALTON-BENCHMARKS/FIBONACCI' │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Coalton development mode without heuristic inlining │
├─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┤
│ Benchmark │ Run time │ Real time │ Bytes consed │ # Iterations │
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤
│ TAK │ 1.043406 s │ 1.044723 s │ 65520 │ 1000 │
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤
│ LISP-TAK │ 0.082777 s │ 0.082867 s │ 65520 │ 1000 │
└─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┘
#.(PACKAGEBENCHMARKRESULTS "coalton-benchmarks/gabriel/tak" #.(COALTON-BENCHMARKING/BENCHMARKING::BENCHMARKSYSTEM "ARM64" "OS-MACOSX" "SBCL" "2.2.4-WIP" COMMON-LISP:NIL COMMON-LISP:NIL) #(#.(BENCHMARKRESULTS "TAK" 1000 1040557 1041583 95888)
#.(BENCHMARKRESULTS "LISP-TAK" 1000 83104 83040 65520)))
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB1 │ 20 │ 0 │ n/a │ 0 │ 1000 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB2 │ 216 │ 0 │ n/a │ 0 │ 1000 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB3 │ 2370 │ 2 │ n/a │ 0 │ 1000 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-GENE │ 17 │ 0 │ 0.0055017565 │ 0 │ 1002 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-GENE │ 95 │ 0 │ 0.0053594956 │ 0 │ 1002 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-GENE │ 912 │ 1 │ 0.0226823009 │ 0 │ 1002 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-GENE │ 10270 │ 10 │ 0.3188679543 │ 0 │ 1014 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-LISP │ 105 │ 0 │ n/a │ 0 │ 1000 │
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
│ REC-FIB-MONO │ 306 │ 0 │ n/a │ 0 │ 1000 │
└───────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘
#.(PACKAGEBENCHMARKRESULTS "COALTON-BENCHMARKS/FIBONACCI" #.(BENCHMARKSYSTEM "ARM64" "OS-MACOSX" "SBCL" "2.2.4-WIP" COMMON-LISP:NIL COMMON-LISP:NIL) #(#.(BENCHMARKRESULTS REC-FIB0 1000 2476 2 #.NONE #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB1 1000 20065 20 #.NONE #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB2 1000 215897 216 #.NONE #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB3 1000 2370219 2370 #.NONE #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-GENERIC0 1002 16914 17 #.(SOME 5.501756508182156d0) #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-GENERIC1 1002 94676 94 #.(SOME 5.359495667149465d0) #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-GENERIC2 1002 911697 910 #.(SOME 22.68230092439423d0) #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-GENERIC3 1014 10270171 10128 #.(SOME 318.8679543261109d0) #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-LISP 1000 104998 105 #.NONE #.(SOME 0))
#.(BENCHMARKRESULTS REC-FIB-MONO 1000 306261 306 #.NONE #.(SOME 0))))
```

`#:run-benchmarks` runs the current package's benchmarks.
Expand Down
Loading

0 comments on commit 4eb3080

Please sign in to comment.