Enhance Error handling

[adhilto]

🗣 Description

• Modify the Provider to track which API calls/function fail
• Modify the Reporter to display which tests errored due to API calls or functions that failed
• Modify the report to check for missing controls and warn the user
• Add missing Rego checks for Gmail

💭 Motivation and context

Closes #157.

Additionally, before this update, if an API call or function failed, the report could display inaccurate information (as evidenced by #147). The terminal would display a warning in cases like this, but it would be better to prevent the inaccuracies from being saved to the report.

The method I implemented for this mirrors the strategy used for ScubaGear.

1. The Provider tracks which API calls/functions succeed/fail
2. The Rego tests are modified to list which API calls/functions they depend on.
3. The Reporter checks to see if the prerequisites were met before displaying the result.

See the following snippet for an example of what step 2 looks like:

tests contains {
    "PolicyId": "GWS.COMMONCONTROLS.7.1v0.1",
    "Prerequisites": ["directory/v1/users/list"],
    "Criticality": "Shall",

At the moment, the vast majority of our tests just depend on the reports API, reports/v1/activities/list, the exceptions being:

• GWS.GMAIL.2.1v0.1
• GWS.GMAIL.3.2v0.1
• GWS.GMAIL.4.1v0.1
• GWS.GMAIL.4.2v0.1
• GWS.GMAIL.4.3v0.1
• GWS.GMAIL.4.4v0.1
• GWS.COMMONCONTROLS.7.1v0.1
• GWS.GROUPS.7.1v0.1

Given that, to reduce code duplication, I made it so that if "Prerequisites" isn't defined for a given test, the Reporter just assumes dependence on reports/v1/activities/list. As Google's API matures, we'll be able to develop more granular prerequisites for each test.

📷 Screenshots (if appropriate)

Examples of what the enhanced error handling looks like in the reports:

🧪 Testing

Note that I did not add the missing checks for group 19, "Attachment Compliance Filtering." That is because that group is being deleted by #156. In the meantime, that means you can run ScubaGoggles and see the new error handling in practice. Once both PRs are merged to main, the warnings will go away.

An easy way to test the error handling is to hard-code an error into the code (like x/0).

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• Changes are limited to a single goal - eschew scope creep!
• If applicable, All future TODOs are captured in issues, which are referenced in the PR description.
• The relevant issues PR resolves are linked preferably via closing keywords.
• All relevant type-of-change labels have been added.
• I have read and agree to the CONTRIBUTING.md document.
• These code changes follow cisagov code standards.
• All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
• Tests have been added and/or modified to cover the changes in this PR.
• All new and existing tests pass.

✅ Pre-merge Checklist

• This PR has been smoke tested to ensure main is in a functional state when this PR is merged.
• Squash all commits into one PR level commit using the Squash and merge button.

✅ Post-merge Checklist

• Delete the branch to clean up.
• Close issues resolved by this PR if the closing keywords did not activate.

[adhilto]

@rgbrow1949 @LaurenBassett Tagging you for awareness, as this is a significant change that everyone should be aware of.

[adhilto]

@buidav @amart241 @snarve this PR is ready for review again. If you need ideas of how to test this, basically break things. Comment out Rego tests, add hard-coded exceptions to provider functions, and see what happens. Just don't commit anything you break :)

[buidav]

Comments

[buidav]

Conditionally approving as the code is functional at the time of my remove. Have a comment below to address before merging.

[LaurenBassett]

Approving these, but definitely want to go over all the changes in our weekly.