2.0 - Test Config - String Formatting

[pflarr]

In combination with variables, string formatting allows us to dynamically insert things into commands and other options in our test sections. We'll use the same example from the #53 (see that ticket for full comments) as a set of variables to demonstrate.

test:
  variables:
    sub-test: 'fs-test'

    options: {mode: '-a', timeout: '30'} 

    # You can give multiple values for a variable. 
    lustre:
      - scratch1
      - scratch2
      - scratch3

    # Variables can be grouped into sub-variables and multiple values.
    filesystems:
      - {type: nfs, path: '/usr/projects/hpctest/'}
      - {type: tmp, path: '/tmp/'}

  # Variables in the permutations, or 'per' section should generally have multiple 
  # values. That's because a virtual test is created and run for each combination 
  # of the values in this section.  
  # In this example, we want four runs, one for each combination of compiler and mpi.
  permutations: 
    mpi: 
      - openmpi
      - mvapich2

    # Sub-variable values are permuted as a group, rather than individually.
    compiler:
      - {name: 'gcc', options: '-O3'}
      - {name: 'intel', options: '-O2'}

Note: all following examples assume the test config also contains the above variable and permutation sections.

Basic insertion

All values in the config are interpreted as strings, and any string in the config can have variables (which include permutation variables) inserted into it.

test:
  run:
    # Now each permutation-generated sub-test will have a reasonable name.
    cmds: './fs-test {sub-test} {options.mode} --timeout={options.timeout}

In this case, we use a variable to specify the sub-test to run. We give one of our options directly, but the other is inserted as the flag value. The key here is that variables are essentially just strings that can be inserted anywhere into your commands.

• Brackets { } indicate a variable to be inserted.
• You can prepend the variable section abbreviation, in case of name conflicts. Otherwise, variables are resolved in the order: var, per, sys (system plugin customizable), pav (pavilion provided).
• While this shares some syntax with the python str.format method, it is a separate implementation and does not support the same formatting features.

Permutation Variables

When you have a multi-valued permutation variable, it creates a virtual test for each. Because of this even if a permutation variable has multiple values, each test only sees the one value it is presented with, and has no access to the others.

test:
  # Create a sub-title to differentiate each virtual test.
  sub-title: '{compiler.name}-{mpi}'

  build:
    # We'll build and run for each compiler/mpi combination.
    modules:
      - {compiler.name}
      - {mpi}
    cmds:
      - '$MPICC -o super_magic super_magic.c'

In this case we build our super_magic test with each combination of compiler and mpi. The $MPICC environment variable would have to be set by either a loaded module or a module wrapper (See #51). Given our config, this would result in four test builds.

Regular Multi-valued Variables

All variable categories other than permutations (var, sys, pav) work pretty much an array or list.

Direct Access

Multi-valued variables may be accessed either explicitly through an index lustre.1, or you can reference the first value implicitly lustre.

test:
  run:
    cmds:
      # The following two are equivalent; they both reference the first value group.
      - './fs_test --type=lustre -p /lustre/{lustre.0}'
      - './fs_test --type=lustre -p /lustre/{lustre}'
      # This works when variables have sub-variables as well.
      - './fs_test --type={filesystems.0.type} -p {filesystems.0.path}'
      - './fs_test --type={filesystems.type} -p {filesystems.path}'

Sub-string Multiples

By putting a sub-string in square brackets [ ], you can cause that sub-string to be repeated for each value of a multi-valued variable.

supermagic:
  run:
    cmds:
      - './super_magic [-w /lustre/{lustre}/] -a'

The command produced would look like:

./super_magic -w /lustre/scratch1/ -w /lustre/scratch2/ -w /lustre/scratch3/ -a 

• If a bracketed sub-string is followed by a space or the end of string, each instance inserted will have a space added as well.

Multiple variables

A square bracketed sub-string can contain have multiple types of variables.

# A more realistic example
supermagic: 
  run: 
    cmds:
      - './super_magic [-w /lustre/{lustre}/{sys.network.name}/$USER] -a'

The resulting cmd would be:

./super_magic -w /lustre/scratch1/prod/$USER -w /lustre/scratch2/prod/$USER -w /lustre/scratch3/prod/$USER -a

You can only have one multi-valued variable in each bracketed sub-string.

More than one multi-valued variable.

As the note above says, you can't do this. You can, however, use a variable with sub-values to contain multiple, related values.

fstest:
  run:
    cmds:
      - './fs_test [-t {filesystem.type} -p {filesystem.path}]'

Would result in:

./fs_test -t nfs -p /usr/projects/hpctest -t tmp -p /tmp/

[pflarr]

This has been implemented and tested.