Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unification benchmark using 'map' and 'stream' #798

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

jiribenes
Copy link
Contributor

@jiribenes jiribenes commented Jan 23, 2025

Resolves #796

It's an OK (not great, not terrible) benchmark:

  1. generates two nested, deep pair types like (...((XLLL, XLLR), (XLRL, XLRR)), ... for X = { A, B },
  2. then unifies them,
  3. and checks that the substitutions are like ALLL -> BLLL.

Configurable with a given N if some tweaking is needed.

Perf:

My machine

On my computer with N = 12:

  • JS compilation only: 4.5s
  • LLVM compilation only: 5.5s
  • then hyperfined:
$ hyperfine --warmup 5 --min-runs 20 './out/unify-js' './out/unify-llvm'
Benchmark 1: ./out/unify-js
  Time (mean ± σ):     703.0 ms ±   7.8 ms    [User: 1035.2 ms, System: 77.1 ms]
  Range (min … max):   690.9 ms … 721.1 ms    20 runs

Benchmark 2: ./out/unify-llvm
  Time (mean ± σ):      37.6 ms ±   0.5 ms    [User: 36.1 ms, System: 1.0 ms]
  Range (min … max):    37.1 ms …  39.7 ms    69 runs

Summary
  ./out/unify-llvm ran
   18.68 ± 0.32 times faster than ./out/unify-js

On my computer with N = 16 (16x more work than N = 12)

  • JS compilation only: 4.7s
  • LLVM compilation only: 5.6s
  • then hyperfined:
Benchmark 1: ./out/unify-js
  Time (mean ± σ):     19.982 s ±  0.608 s    [User: 26.011 s, System: 1.525 s]
  Range (min … max):   19.300 s … 21.267 s    20 runs

Benchmark 2: ./out/unify-llvm
  Time (mean ± σ):     928.2 ms ±  20.8 ms    [User: 906.3 ms, System: 16.1 ms]
  Range (min … max):   913.3 ms … 1006.2 ms    20 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  ./out/unify-llvm ran
   21.53 ± 0.81 times faster than ./out/unify-js

CI

With N = 12 in CI (measured imprecisely in two CI rounds):

  • 9.2-12.0 seconds on Chez backends
  • 8.0-8.9 seconds on JS
  • 9.9-12.7 seconds on LLVM

With N = 16 in CI (measured imprecisely in two CI rounds):

  • 21-29 seconds on Chez backends
  • 55-57 seconds on JS
  • 79-86 seconds on LLVM

@jiribenes jiribenes added bug Something isn't working area:stdlib area:tests labels Jan 23, 2025
@jiribenes jiribenes marked this pull request as draft January 23, 2025 16:03
@jiribenes

This comment was marked as resolved.

@jiribenes
Copy link
Contributor Author

jiribenes commented Jan 23, 2025

Here's a slightly minified version:

/// Robinson-style Unification Algorithm
module examples/benchmarks/unify

import map
import result
import bytearray

type Type {
  Var(name: String)
  Con(name: String, args: List[Type])
}

type Substitution = Map[String, Type]

type UnificationError {
  OccursCheckFailure(variable: String, tpe: Type)
  UnificationFailure(tpe1: Type, tpe2: Type)
  UnificationManyFailure(tps1: List[Type], tps2: List[Type])
}

def showType(ty: Type): String = ty match {
  case Var(name) => name
  case Con(name, Nil()) => name
  case Con(name, args) =>
    name ++ "(" ++ args.map { t => showType(t) }.join(", ") ++ ")"
}

def show(err: UnificationError): String = err match {
  case OccursCheckFailure(variable, ty) =>
    "Occurs check failed: " ++ variable ++ " occurs in " ++ showType(ty)
  case UnificationFailure(ty1, ty2) =>
    "Cannot unify " ++ showType(ty1) ++ " with " ++ showType(ty2)
    case UnificationManyFailure(tps1, tps2) =>
    "Cannot unify " ++ tps2.map { t => showType(t) }.join(", ") ++ " with " ++ tps1.map { t => showType(t) }.join(", ")
}

def reporting { body : => Substitution / Exception[UnificationError] }: Unit = {
  val res = result[Substitution, UnificationError] {
    body()
  }

  res match {
    case Success(subst) => {
      println("Unification successful!")
      println("Substitution:")
      subst.foreach { (k, v) =>
        println("  " ++ k ++ " -> " ++ showType(v))
      }
    }
    case Error(err, msg) =>
      println("Unification failed: " ++ show(err))
      if (msg.length > 0) {
        println(msg)
      }
  }
}

def main() = reporting {
  val intType = Con("Int", [])
  val listType = Con("List", [intType])
  val typeVar = Var("a")

  do raise(UnificationManyFailure([intType], [listType, typeVar]), "")
}

seems to crash the normaliser!

Exception in thread "main" java.lang.StackOverflowError
        at effekt.symbols.Symbol.equals(Symbol.scala:37)
        at effekt.symbols.Symbol.equals$(Symbol.scala:18)
        at effekt.symbols.TrackedParam$BlockParam.equals(symbols.scala:124)
        at scala.runtime.BoxesRunTime.equals2(BoxesRunTime.java:133)
        at scala.runtime.BoxesRunTime.equals(BoxesRunTime.java:119)
        at scala.collection.immutable.Map$Map4.contains(Map.scala:516)
        at effekt.core.Renamer$$anon$2.applyOrElse(Renamer.scala:46)
        at effekt.core.Renamer$$anon$2.applyOrElse(Renamer.scala:45)
        at scala.collection.IterableOnceOps.collectFirst(IterableOnce.scala:1116)
        at scala.collection.IterableOnceOps.collectFirst$(IterableOnce.scala:1108)
        at scala.collection.AbstractIterable.collectFirst(Iterable.scala:933)
        at effekt.core.Renamer$$anon$1.applyOrElse(Renamer.scala:46)
        at effekt.core.Renamer$$anon$1.applyOrElse(Renamer.scala:44)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:441)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:445)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:444)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:60)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:50)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:444)
        at effekt.core.Renamer$$anon$3.applyOrElse$$anonfun$4(Renamer.scala:61)
        at effekt.core.Renamer.withBindings(Renamer.scala:37)
        at effekt.core.Renamer.withBinding(Renamer.scala:41)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:61)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:50)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:444)
        at effekt.core.Renamer$$anon$4.applyOrElse$$anonfun$9(Renamer.scala:78)
        at effekt.core.Renamer.withBindings(Renamer.scala:37)
        at effekt.core.Renamer$$anon$4.applyOrElse(Renamer.scala:79)
        at effekt.core.Renamer$$anon$4.applyOrElse(Renamer.scala:74)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:445)
        at effekt.core.Renamer$$anon$3.applyOrElse$$anonfun$2(Renamer.scala:53)
        at effekt.core.Renamer.withBindings(Renamer.scala:37)
        at effekt.core.Renamer.withBinding(Renamer.scala:41)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:53)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:50)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:444)
        at effekt.core.Renamer$$anon$3.applyOrElse$$anonfun$2(Renamer.scala:53)
        at effekt.core.Renamer.withBindings(Renamer.scala:37)
        at effekt.core.Renamer.withBinding(Renamer.scala:41)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:53)
        at effekt.core.Renamer$$anon$3.applyOrElse(Renamer.scala:50)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:444)
        at effekt.core.Renamer$$anon$4.applyOrElse$$anonfun$9(Renamer.scala:78)
        at effekt.core.Renamer.withBindings(Renamer.scala:37)
        at effekt.core.Renamer$$anon$4.applyOrElse(Renamer.scala:79)
        at effekt.core.Renamer$$anon$4.applyOrElse(Renamer.scala:74)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at effekt.core.Tree$Rewrite.rewrite(Tree.scala:453)
        at effekt.core.Renamer$.rename(Renamer.scala:111)
        at effekt.core.optimizer.Normalizer$.reduce(Normalizer.scala:378)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:176)
        at effekt.core.optimizer.Normalizer$.normalizeVal$1(Normalizer.scala:215)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:283)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:303)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:308)
        at effekt.core.optimizer.Normalizer$.active(Normalizer.scala:126)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:164)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:165)
        at effekt.core.optimizer.Normalizer$.reduce(Normalizer.scala:387)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:176)
        at effekt.core.optimizer.Normalizer$.normalizeVal$1(Normalizer.scala:215)
        at effekt.core.optimizer.Normalizer$.normalize(Normalizer.scala:283)

When I do --no-optimize, this minified version produces the correct results.

Co-authored-by: Jonathan Brachthäuser <[email protected]>
@jiribenes jiribenes changed the title WIP: Unification benchmark using 'map' Unification benchmark using 'map' and 'stream' Jan 23, 2025
@jiribenes jiribenes marked this pull request as ready for review January 23, 2025 19:41
@jiribenes jiribenes requested a review from b-studios January 23, 2025 20:09
Comment on lines +159 to +160
// the actual test
val N = 12
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

N = 12 means unification of two trees of depth 12, i.e. tree of tuples with 2^12 = 4096 variables in the leaves.
We can tweak this number or even use the benchmarks/runner...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:stdlib area:tests bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a benchmark involving 'map' and/or 'set'
2 participants