1.14.0

1.14.0 (2023-12-11)

Features

Change the default output to be scaled to the inner loop count (and for throughput, always scale to this)

Add a new metric peakMemoryResidentDelta that discounts any test setup memory footprint and only shows the delta for the actual benchmark run for RSS.

What's Changed

Fixed TSAN failure
Fix (sometimes) cut off description
Proper checks for missing absolute threshold paths

• feat(minor): Various outstanding fixes by @hassila in #204

Full Changelog: 1.13.0...1.14.0

1.13.0

1.13.0 (2023-11-28)

Features

• minor: Lower required swift-system version to 1.1 to facilitate benchmark adoption for more users (#203) (f629323)

1.12.0

1.12.0 (2023-11-14)

Bug Fixes

• Add missing tag in Influx datatype header (#197) (d632894)

What's Changed

• chore: update package dependencies by @github-actions in #196
• docs: Remove outdated section by @MahdiBM in #199
• fix: Add missing tag in Influx datatype header by @clackary in #197
• chore(minor): Fix sampling for virtual/resident memory peaks by @hassila in #200

New Contributors

• @MahdiBM made their first contribution in #199
• @clackary made their first contribution in #197

Full Changelog: 1.11.2...1.12.0

1.11.2

1.11.2 (2023-09-22)

1.11.1

1.11.1 (2023-09-14)

Bug Fixes

• patch: Show exact deviations if rounded are zero (#188) (6e0cdf8)

1.11.0

1.11.0 (2023-09-14)

Features

Its desirable to be able to signal from CI if a threshold has been improved too and not only regressed, so we now return exit code 0 only for exact equal checks and 2 for regressions and 4 for improvements for baseline checks and absolute threshold comparisons.

Added support for signposts (#183) which makes profiling from Instruments easier to analyze.

The easiest way to access Instruments and profile a benchmark, is to simply quit Xcode and open the package with:
open --env BENCHMARK_DISABLE_JEMALLOC=1 Package.swift
This works around the Xcode bug that makes it impossible to use jemalloc and one can simply build for Profile and use instruments as usual - now with signpost information per benchmark run:

Added new objectAllocCount ARC stat counter and fixed the delta calculation to also take this initial refcount into account so retain/release deltas should be closer to reality (still sometimes non-zero due to some missing runtime hooks).

Also added support for optionally returning setupState from the benchmark setup hooks, to allow access to state that can be reused per benchmark iteration which is provided as an additional second parameter to the benchmark closure.

This can easily be done like this:

    Benchmark("SetupTeardown6") { benchmark, setupState in
        print("Array of integers: \(setupState)")
    } setup: {
        [1, 2, 3]
    }

Bug Fixes

• minor: Add support for CI detection of absolute threshold improvements + signposts for Instruments (#182) (1e3568b)

• minor: Fix incorrect syscalls (#187) (8e6ffd9)

1.9.2

1.9.2 (2023-09-13)

Bug Fixes

• patch: quote test names with comma in output CSV (#185) (d7a1343)

1.9.1

What's Changed

• feat(patch): support bytesRead/bytesWritten on macOS by @dimlio in #181

Full Changelog: 1.9.0...1.9.1

1.9.0

1.9.0 (2023-08-24)

This provides better out-of-the-box experience for absolute checks and is geared towards CI checking of e.g. malloc/syscall deviations. The thresholds can e.g. be put be in a "Thresholds" directory.

To generate a new baseline, run e.g.

swift package --allow-writing-to-package-directory benchmark --format metricP90AbsoluteThresholds --path Thresholds/

To run a check vs. the saved thresholds, run:

swift package benchmark baseline check --check-absolute-path /relative/or/absolute/path/to/Thresholds

Features

• minor: Add support for metricP90AbsoluteThresholds export format (#180) (3549c2b)

1.8.2

1.8.2 (2023-08-22)

Bug Fixes

• patch: Use @_optimize(none) instead of @inline(never) for blackHole to ensure it's not broken by cross-module optimisation in Swift 5.8 (#179) (82cd144)