ISPC performance tracking

[nurmukhametov]

Hi! I am an ISPC developer, and I would like to add openmoonray build into CI to track the correctness and performance of ISPC compiler on your codebase (the biggest known to me in the open-source). I have two questions:

1. How do you check the correctness (ISPC code, some parts of openmoonray, rendering as whole)?
2. Which benchmarks or examples are better in regard to performance measurement of ISPC code?

[randypacker]

Hi @nurmukhametov,

1. How do you check the correctness (ISPC code, some parts of openmoonray, rendering as whole)?

We have just a few unit tests that directly compare ISPC/C++ for correctness, and there is a lot of room for improvement in our coverage.  See for example, this small test for the snapshotDelta logic.  Note that it's currently a bit fragile and will be updated in the next release.  

We also run a suite of simple scenes designed to test each feature as our continuous integration tests (called "RATS") that compare rendered output to canonical images, and flag look discrepancies beyond a certain pixel threshold.  We are working on adding a simplified mini regression test suite to OpenMoonRay in an upcoming release, and plan on extending it by migrating some of our internal tests over time.

2. Which benchmarks or examples are better in regard to performance measurement of ISPC code?

We run a suite of diverse production scenes from our films with each of our weekly internal releases, and use the Render Profile Viewer to parse the logs and plot the resulting stats in a graph so we can visually spot issues and help identify which areas of the code are likely the culprit for any performance regressions, or benefit from expected performance improvements.  We do these runs in scalar, vector and xpu modes.

Hope this helps!