2.0 - Pluggable Result Parsing

[pflarr]

After a test runs to completion, the stdout of that test should be written to a <results_dir>/run.out file. Pav should provide a series of plugins that can be selected and configured to parse those results. This ticket covers the creation of a result parser plugin system using yapsy. Child tickets should cover the creation of parser plugins for common situations. Like scheduler plugins, these will add a unique configuration section to test configs. Unlike schedulers, only one of these sections should be allowed in a test config.

The results of parsing should be a JSON mapping. The structure of the keys under this mapping is up to the individual parser plugin, but the following keys are reserved:

• start_time
• end_time
• user
• result

[nicholas-sly]

Parsing of the results should provide some options to the users for extracting specific relevant results to determine the final result for the whole run.

Workable methods of extracting include (both of which allows for lists of these):

• Specify an exact string to search for
• Specify a regular expression to match against
• Specify results for a given return code

Types of results can include:

• Unqualified Success - The test ran and passed any and all tests.
• Partial Success - A series of tests were searched for and only some came back positive.
• Value-based Failure - Values extracted were compared to a standard for success and failed.
• Failure - All tests were failed.
• Failed to complete - Failed before completing the test itself (failed during build, setup, or execution phases).

Value-based tests:

• Min/Max value - Values specified in the pavilion configuration YAML as maximums or minimums. Requires a method of explaining how to extract the numeric support and what format to expect.
• Self-contained statistical tests - Specify a statistical test for a series of values produced by the test. An example includes every node individually performing some task within some error of the mean wall time.
• Centralized statistical tests - Testing that the extracted result passes some statistical test based on values stored in a central database. An example would be achieving a similar result to previous runs of the same test.

Tests for partial success:

• Series of value-based tests: Some places where the result was found did not pass the value-based test, but others did.
• Range-based tests: A result was given standards for success, partial-success, and failure (ranges of values) where the result fell in the partial-success range but not in the success range.

The types of successes should be plugins as well as the types of parsing methods.

[pflarr]

I think we should keep the pass/fail results simple.

• Success
• Failures
• Error (with a separate error type (build, scheduling, runtime)

[nicholas-sly]

We can make that a 'basic' plugin for the results parsing and then allow for other plugins that do more.

[pflarr]

We can have result parsers that grab and put all kinds of data into the result dictionary, but the result value itself should be simple; one of a few fields that let you glom the results of all tests together in a logical way in Splunk. Pass/Fail/Error are about as complicated as you can get without forcing the user to dig into test configs to see what a result actually means. While a Success and Fail are inherently arbitrary too, a "Partial Success" really begs the question of what that actually means for a given test.

Instead, I'd just say 'Success', and then include a separate field that shows the sub-test pass fail rate or distance from expected values for that particular test.