Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nice things (test refactor) from PR 39 #42

Merged
merged 7 commits into from
Jun 24, 2019
Merged

Conversation

warpech
Copy link
Contributor

@warpech warpech commented Jun 24, 2019

PR #39 contains discussable changes that downgrade performance, but also refactoring of the test suite.

Since I don't want to merge performance degradation until absolutely required, I decided to make another PR with things that are easier to agree with.

In fact, these changes were already agreed, because PR #39 got a green light for merging. Just in case, @tomalec please give a quick review again. Changes are only in package.json and tests.

Changes are explained in the commit messages, but here's a short summary:

  • DRY-er unit tests and benchmark suites
  • more meaningful benchmark suites (e.g. for getting data from the observed object)
  • old command npm run bench does not compare with fast-json-patch anymore
  • new command npm run bench-compare compares with fast-json-patch and no-op

I think this can be merged to master before PRs #39 and #41, then I can open new rebased PRs for #39 and #41 (this won't take long).

Benchmark formatting before this PR:

jsonpatcherproxy generate operation
 Ops/sec: 144269 ±2.90% Ran 9277 times in 0.064 seconds.
jsonpatch generate operation
 Ops/sec: 97748 ±7.24% Ran 6095 times in 0.062 seconds.
jsonpatcherproxy mutation - huge object
 Ops/sec: 790 ±3.76% Ran 50 times in 0.063 seconds.
jsonpatch mutation - huge object
 Ops/sec: 761 ±2.86% Ran 45 times in 0.059 seconds.
jsonpatcherproxy generate operation - huge object
 Ops/sec: 708 ±4.18% Ran 46 times in 0.065 seconds.
jsonpatch generate operation - huge object
 Ops/sec: 851 ±1.40% Ran 47 times in 0.055 seconds.
PROXIFY big object
 Ops/sec: 1805 ±2.05% Ran 104 times in 0.058 seconds.
DIRTY-OBSERVE big object
 Ops/sec: 1697 ±1.17% Ran 93 times in 0.055 seconds.

Benchmark formatting after this PR:

Observe and generate, small object (noop)
 Ops/sec: 12434413 ±0.39% Ran 637179 times in 0.051 seconds.
Observe and generate, small object (JSONPatcherProxy)
 Ops/sec: 95511 ±2.74% Ran 5983 times in 0.063 seconds.
Observe and generate, small object (fast-json-patch)
 Ops/sec: 68608 ±1.63% Ran 3780 times in 0.055 seconds.
Observe and generate (noop)
 Ops/sec: 164602 ±0.30% Ran 8453 times in 0.051 seconds.
Observe and generate (JSONPatcherProxy)
 Ops/sec: 1893 ±1.92% Ran 204 times in 0.108 seconds.
Observe and generate (fast-json-patch)
 Ops/sec: 1650 ±0.98% Ran 95 times in 0.058 seconds.
Primitive mutation (noop)
 Ops/sec: 2601631 ±0.32% Ran 132729 times in 0.051 seconds.
Primitive mutation (JSONPatcherProxy)
 Ops/sec: 524799 ±0.19% Ran 26797 times in 0.051 seconds.
Primitive mutation (fast-json-patch)
 Ops/sec: 7955 ±0.36% Ran 409 times in 0.051 seconds.
Complex mutation (noop)
 Ops/sec: 6382115 ±0.23% Ran 325296 times in 0.051 seconds.
Complex mutation (JSONPatcherProxy)
 Ops/sec: 8731 ±0.30% Ran 448 times in 0.051 seconds.
Complex mutation (fast-json-patch)
 Ops/sec: 8281 ±0.26% Ran 424 times in 0.051 seconds.
Serialization (noop)
 Ops/sec: 7036 ±0.38% Ran 365 times in 0.052 seconds.
Serialization (JSONPatcherProxy)
 Ops/sec: 1278 ±0.34% Ran 66 times in 0.052 seconds.
Serialization (fast-json-patch)
 Ops/sec: 7092 ±0.49% Ran 365 times in 0.051 seconds.

warpech added 7 commits June 24, 2019 14:10
use a one liner to clone the object where possible
Warning, this commit changes the performance of the benchmark. Generation of tested object is now 2x faster. This is good for the benchmark, because cheaper object generation allows the benchmark to be more sensitive in catching the differences in the performance of JSONPatcherProxy.

Just remember that the results from before and after this commit are not comparable.

Benchmark results before this commit
=============================
jsonpatcherproxy generate operation
 Ops/sec: 132404 ±2.55% Ran 8276 times in 0.063 seconds.
jsonpatch generate operation
 Ops/sec: 106585 ±6.73% Ran 6434 times in 0.060 seconds.
jsonpatcherproxy mutation - huge object
 Ops/sec: 906 ±2.45% Ran 51 times in 0.056 seconds.
jsonpatch mutation - huge object
 Ops/sec: 862 ±1.52% Ran 47 times in 0.055 seconds.
jsonpatcherproxy generate operation - huge object
 Ops/sec: 811 ±1.99% Ran 46 times in 0.057 seconds.
jsonpatch generate operation - huge object
 Ops/sec: 847 ±1.20% Ran 47 times in 0.055 seconds.
PROXIFY big object
 Ops/sec: 1813 ±2.00% Ran 102 times in 0.056 seconds.
DIRTY-OBSERVE big object
 Ops/sec: 1625 ±1.48% Ran 91 times in 0.056 seconds.

Benchmark results after this commit
=============================
jsonpatcherproxy generate operation
 Ops/sec: 136915 ±2.46% Ran 8566 times in 0.063 seconds.
jsonpatch generate operation
 Ops/sec: 104113 ±7.19% Ran 6330 times in 0.061 seconds.
jsonpatcherproxy mutation - huge object
 Ops/sec: 1960 ±3.53% Ran 130 times in 0.066 seconds.
jsonpatch mutation - huge object
 Ops/sec: 1607 ±1.84% Ran 88 times in 0.055 seconds.
jsonpatcherproxy generate operation - huge object
 Ops/sec: 1562 ±3.40% Ran 101 times in 0.065 seconds.
jsonpatch generate operation - huge object
 Ops/sec: 1620 ±2.04% Ran 90 times in 0.056 seconds.
PROXIFY big object
 Ops/sec: 3968 ±4.07% Ran 294 times in 0.074 seconds.
DIRTY-OBSERVE big object
 Ops/sec: 3285 ±1.83% Ran 180 times in 0.055 seconds.
the benchmark methods are almost exactly the same, but I have made the following changes:
- group benchmarks by the topic
- give more meaningful titles
- wherever the preparation code is not meaningful for the benchmark, move it outside the benchmark
- use code blocks to give scope to preparation variables

this is a change that makes the benchmark results incomparable with the previous versions.

however, it will work if you copy this file onto an older version and run it.
as requested in #39 (comment)

the benchmark simply serializes the whole object, which triggers the "get" trap on every subtree
by default such comparison will be excluded. Benefits of this:
- faster execution
- shorter report to copy to PRs
to make functionality equal to fast-json-patch'es
Copy link
Member

@tomalec tomalec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks ❤️

@warpech warpech merged commit fbb8210 into master Jun 24, 2019
@tomalec tomalec deleted the nice-parts-from-pr-39 branch June 24, 2019 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants