8. Conditional Test Operators

Introduction

1. Conditional operators introduced in v1.1:

   • AND
   • OR
   • NOT

2. Following grammar is used by the operators:

   exp -> catA-op exp exp | catB-op ETC

   catA-op -> AND | OR

   catB-op -> NOT | epsilon

   ETC: Elementary test condition

3. AND, OR are n-ary operators. NOT is an unary operator. If any deviation is observed, for ex, passing more than one ETC to a unary operator or passing one ETC to a n-ary operator, the parser will simply ignore that logical sub-expression and continue evaluating remaining sub-expressions.

4. If all sub-expressions are skipped then no ETCs are executed. For module version, this means that Operator.result will now be None instead of earlier True.

5. Refer to Example 2 below. If instead of the top most OR operator you want to use AND operator, there is no need to write it explicitly. The parser will consider AND to be the default operator will evaluating. Thus one can write:

   tests:
        - AND:  
            - is-equal: admin-status, up
            - is-equal: admin-status, down
        - OR:  
            - is-equal: oper-status, up
            - is-equal: oper-status, down

   It will be evaluated as (? AND ?) AND (? OR ?)

6. [IMP] The operators have evaluation shortcut optimization builtin. This means that the evaluation of the subexpression under AND operator will stop when the first false is encountered. Remaining subexpressions down the list will be skipped. In case of OR operator when the first true is encountered, evaluation of that subexpression is stopped. Refer example for in-depth explanation.

[IMP] Changes to reporting

1. Nothing is removed only new things are added.
2. Each test's status is also shown in the final report. Here each test is treated as an expression and the result of evaluation of that expression is reported.
3. For module version, one can access the new Operator.result_dict dictionary (key => test_name, value=> result[True/False/None]) to access the result of each test. User should ensure that test names in a test suite remain unique. If they repeat, the result of older instance gets overwritten.
4. One should not confuse the no. of test cases passed or fail reported at the end with the no. of sub-expressions/expressions failed or passed.

Examples

Example 1

test.yml

tests_include:
- test_interfaces_terse

test_interfaces_terse:
- command: show interfaces terse lo* 
- item:
   id: ./name
   xpath: //physical-interface[normalize-space(name) = "lo0"]
   tests:
     - AND:  
         - is-equal: admin-status, down
           info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
           err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
         - is-equal: admin-status, up
           info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
           err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output

Explanation

1. There were two ETCs namely - is-equal: admin-status, down and - is-equal: admin-status, up.

2. As reported in the results the first ETC passed and the second failed.

3. The result of this AND(pass, fail) = fail is reported in the final result in the following way

   test_interfaces_terse : Failed i.e. <test_name> : <result of expression evaluation>

4. "Total number of tests" in the final result mean the total number of ETCs

5. Notice that the final result is "Failed".

Example 2

test.yml

tests_include:
- test_interfaces_terse

test_interfaces_terse:
- command: show interfaces terse lo* 
- item:
   id: ./name
   xpath: //physical-interface[normalize-space(name) = "lo0"]
   tests:
     - OR:
         - AND:  
             - is-equal: admin-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: admin-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
         - OR:  
             - is-equal: oper-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: oper-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output

Explanation

1. There are total of 4 ETCs.
2. As one can see, only two ETCs were run.
3. In the AND subexpression, the first ETC (i.e. - is-equal: admin-status, up) evaluated to False and hence the second ETC (i.e. - is-equal: admin-status, down) was not evaluated as whatever may be the result, the final result of False AND ? is False.
4. In the second OR subexpression, the first ETC evaluated to True and hence the second ETC was not evaluated as True OR ? is True.
5. The final result of the the test came out to be Passed as (False AND ?) OR (True OR ?) is TRUE.

Example 3

test.yml

tests_include:
- test_interfaces_terse

test_interfaces_terse:
- command: show interfaces terse lo* 
- item:
   id: ./name
   xpath: //physical-interface[normalize-space(name) = "lo0"]
   tests:
     - OR:
         - AND:  
             - NOT:
               - is-equal: admin-status, up
                 info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
                 err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               - is-equal: admin-status, down
                 info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
                 err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
         - OR:  
             - is-equal: oper-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: oper-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output

Explanation

1. Please not that the NOT and AND subexpression is illegal and hence we have an error of malformed subexpression.
2. In such scenarios, the parser tries to recover as much as possible.
3. In this case, whole of the AND subexpression is skipped and the remaining expression:

   tests:
     - OR:
        - OR:  
             - is-equal: oper-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: oper-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
   

   is evaluated.