2.0 - Test Configs

[pflarr]

Note: For the following, a 'test section' isn't the whole test yaml file like we're used to. It is the section of the config file that describes each test. These currently exist under a 'test name' key, and will continue to do so.

The following features will be added to test configuration:

1. Test configuration consists of several layers, each of which overrides the previous.
   a. A 'host' config containing a single test section for configuring things specific to that host.
   b. Zero or more 'mode' configs that each contain a single test section, for setting various general running modes and scheduler settings.
   c. A test config from a test config file containing one or more test sections in a yaml mapping.
2. A strict definition of the components of a test section.

• This will be checked before a test series is kicked off.
• Various plugins will add to it dynamically.

3. In test configs, the key value for a test section will be that test's name.
4. The test 'run' section will be improved to allow for dynamically created test run scripts. (See 2.0 - Test Config - Run #50)
5. Dynamic string formatting (See 2.0 - Test Config - String Formatting #54 )
6. Variable section (See 2.0 - Test Config - Variables #53 )
7. Test Inheritance (See 2.0 - Test Config - Inheritance #55 )
8. Test 'list multiplication' will only be allowed within the variable section, to clarify configs and simply the format.

[jennfshr]

* Test_args will be removed This will break existing tests that play heavily on the existence of test_args… Jen

[pflarr]

Yes, and no.
It will break existing test yaml files, but it won't break runit scripts. Those can still be run with the same args (and permutations of args), but through string substitution instead. The amount of editing needed for the conversion is fairly minimal.

[pflarr]

Also, having multiple cmds per test is necessary for our design goals of reducing the need for runit scripts altogether, and that can't coexist with test_args in any reasonable way that I can think of.

[jennfshr]

Elaborate on string substitution as a replacement for PV_TEST_ARGS?

[pflarr]

I do believe I was mistaken about what test_args was actually doing. I thought they were getting appended to the command, but I guess they were just getting set as an environment variable? Maybe both?

Regardless, under the new config schema, you would simply set PV_TEST_ARGS as an environment variable for the test run. The old runit scripts shouldn't break in that case.

As for the string substitution, we're still settling on what that will look like. Both Nick and I have very different proposals, that are being discussed.