Matrix testing to ensure business coverage of rules

[marcaurele]

Hi,

I have a question on how to best write the test cases for our rules as it tends to grow with way too many repetition today.

Currently we are writing the tests like:

roles := {"000-001": "role-1", "000-003":"role-2"}

test_action_a_path_in {
    input_data = {
        "action": "service:module:action-a",
        "roles": roles,
        "path": ["000-0001", "003-001", "023-001"]
    }
    data.authorization.v1.allow with input as input_data
}

test_action_a_path_out {
    input_data = {
        "action": "service:module:action-a",
        "roles": roles,
        "path": ["000-0002", "003-002", "023-002"]
    }
    data.authorization.v1.allow with input as input_data
}

test_action_b_path_in {
    input_data = {
        "action": "service:module:action-b",
        "roles": roles,
        "path": ["000-0001", "003-001", "023-001"]
    }
    data.authorization.v1.allow with input as input_data
}

test_action_b_path_out {
    input_data = {
        "action": "service:module:action-b",
        "roles": roles,
        "path": ["000-0002", "003-002", "023-002"]
    }
    data.authorization.v1.allow with input as input_data
}

So basically we repeat all the test to cover the matrix of action x path possibilities. This is to ensure the definition of the rules was correctly defined going over all the possible cases. We are reaching 3000 tests today.

Is there a way to create an easy parametrized list of test, like what I can refer to with pytest?

How would be a best approach to write this test matrix covering all the possible cases?

[anderseknert]

Hi @marcaurele 👋

There isn't currently an "official" way to do parameterized tests, although it's been on my wish list for a long time. See this issue and the discussion there for some possible workarounds. And of course, make sure to upvote it! And add a comment if you have ideas on what you'd want it to look like.

An interesting take I saw on this just some week ago was in the CVAT project, where they generate Rego tests from CSV data to cover all combinations (over 30K tests!): https://github.com/opencv/cvat/tree/develop/cvat/apps/iam/rules

[marcaurele]

I thought about generating the test files but I hopped there would be a better alternative 😉 - Thanks for the heads up on the feature request!