Proposal: add a field to the VSA schema for policy violations

[djtjwillia]

Currently, the VSA schema shows the policy bundle, and a verificationResult of PASSED or FAILED. Today, based on the VSA, there is no way for consumers of an artifact to determine if they would like to make an exception to use an artifact with a FAILED verifactionResult, as there is no context in the VSA about what violations were found. Similarly, from the VSA, producers do not have any actionable listing on what caused a FAILED state to be determined.

If we add an optional field, policyViolations, that would be a collection of name, description pairs that could be output by the policy evaluator to provide detail on this policy violations then either consumers or producers could address or use FAILED policy artifacts with a higher level of ease and confidence. This would also benefit those who may currently be creating custom attestations specifically for policy results.

[trishankatdatadog]

Thanks for opening this issue, Taylor!

@TomHennen @adityasaky What is our current thinking on failure modes for VSA again? Do we want a VSA that says a policy failed to verify and why exactly? I think so: I can imagine use cases where that is very useful (e.g., a latest failed ticket that supersedes a previous successful one).

[TomHennen]

Just getting back from Christmas break...

It sounds like you have two use cases:

1. Let people know what the failures were so they can be fixed.
2. Let people know what the failures were so they can be ignored and the artifact can be used anyways.

I understand 1 well enough, 2 I'm somewhat concerned about. Can you say more about what your trying to accomplish?

For context, the way we use VSAs is something like:
a. (ideally) If a verification fails, the artifact, and its VSA, are not published, so no one should be in a position to use a failed artifact.
b. When verification fails, the service that does the verification and issues the VSA, returns error messages and other signals that are displayed to the project owner so that they can fix any issues.
c. Properties from the policy that succeed are stored in the verifiedLevels field of the VSA. Users that want to ensure a specific property was checked can look in that field to see if did happen.
d. In the case that artifacts are still published even if verification fails, we send the VSA to our internal inventory to record the fact of the failure, and we do include error messages. However we don't typically want to publish this VSA as those failure messages could be sensitive, and could be used for cases like (2) which I think are dangerous.

Generally we use VSAs as a means of decoupling the policy itself from downstream users. Exposing downstream users to those details would couple those cases and could lead to both integration challenges and security issues.

In this case specifically, imagine if a policy added a new, very important check, that check fails, downstream users are ignoring the 'FAILED' status and only looking for specific types of failures. They don't know about this new check, so they use the artifact, and then they get owned (or whatever :) ). If downstream users are told "if the status is FAILED don't use the thing" it's a much easier message.

if the question is just one of debugging (1 above) then the RPC (b) will return the information the owner needs to determine what should be done.

So, that being said, can you say more about the problem you're trying to solve? I wonder if it can be addressed at other parts of the system?

[adityasaky]

+1 to more information about the problem that'd be addressed. 😄

I think communicating problems so that they can be fixed may not be worth using a VSA for. I suspect attestations aren't expected to be very human readable and there are probably friendlier ways of communicating the problems than in the attestation.

The second case of using that information in applying policy against the VSA is an interesting one, but has the downsides Tom mentioned above.

More generally, I wonder if this information, if we decide to record as an attestation at all, belongs in a separate type altogether for verification failures. I personally prefer if a VSA is only generated when verification is successful (i.e., result is always PASSED or removed altogether). This is easier to check ("success is when evaluator sees VSA" instead of 'success is when evaluator sees VSA and verificationResult == PASSED") and thus less error prone. Using a separate type to handle failure scenarios that may still be fine possibly makes this easier to reason about and harder to accidentally use the attestation to meet an expected success case. Also curious to hear @marcelamelara's thoughts on this one!

[marcelamelara]

Interesting discussion! Thanks for the ping @adityasaky .

As others have mentioned, I'd be curious to hear more details about what the goal is of supporting failure exceptions in VSAs. It's not entirely unprecedented to make exceptions for certain predicates: Macaron, for example, supports exceptions to the policy, as I understand it (though I'm fuzzy on the details), but Macaron operates at a pretty fine level of granularity, so it allows for the exceptions to be programmed into the policy checks. This is related, but separate, from what this discussion is about.

So, I generally agree with the direction that @adityasaky is going, which is to clarify the semantics of the VSA. Should the existence of a VSA always imply a PASSED status for policy checks (i.e. there is no status field in the VSA)? This seems to have been our assumption so far, and we haven't really articulated as clearly (in the spec) how a consumer should handle a FAILED status.

If debugging is a primary goal, I also agree that VSAs aren't the right place to record this information. If verification of an artifact is being outsourced to a trusted third party, a separate predicate for policy errors (or some existing one) can be used
to authenticate the policy checker's error logs. If the intent is to determine different confidence levels in an artifact, the SLSA track levels already listed in the VSA serve to aid in this decision.