Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks should check consistent results #5

Open
dmah42 opened this issue Jun 5, 2013 · 1 comment
Open

Benchmarks should check consistent results #5

dmah42 opened this issue Jun 5, 2013 · 1 comment

Comments

@dmah42
Copy link
Owner

dmah42 commented Jun 5, 2013

Currently, benchmarks exercise behaviour but do not check whether the calculations are correct (consistent with previous runs).
Ideally, the benchmarks might be switched to unit tests but a start would be having the program spot and highlight deviations.

@dmah42
Copy link
Owner Author

dmah42 commented Jun 5, 2013

So began investigating...

For Box2D (C++), the closest to the demos we have are in the Testbed app (which uses .h files as loadable 'levels'). Don't see and checksums here though (yet)?

For JBox2D (Java), there are identically named demos in jbox2d-testbed, e.g:
jbox2d-testbed/src/main/java/org/jbox2d/testbed/tests/DominoTest.java
jbox2d-testbed/src/main/java/org/jbox2d/testbed/tests/DominoTower.java

However, (a) "mvn install" is failing for me (still investigating) and (b) there's again no mention of checksums!?
(grep -ri checksum in C++ and Java repos find nothing. benchmark finds nothing in C++ and Java finds "jbox2d-testbed/src/main/java/org/jbox2d/testbed/perf/" which includes some values in comments.)

Searching, I think the answer to this may be "there is no such thing as consistency between platforms" per this link from the FAQ:
http://code.google.com/p/box2d/wiki/FAQ#Is_Box2D_deterministic?

As such, is it sufficient (for now) to strive for intra-Dart consistency (definitely between fix-versions and hopefully between Dartium, DartVM and dart2js versions in Chrome and command-line V8)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant