Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENGPROD-6654] Ignore policies from a folder #6

Merged
merged 3 commits into from
Feb 15, 2024
Merged

[ENGPROD-6654] Ignore policies from a folder #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9b1482d
Ignore policies from a folder
dstrelbytskyi Feb 14, 2024
6d6d77e
Updated error messages.
dstrelbytskyi Feb 15, 2024
a27860c
Added debug logging
dstrelbytskyi Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/flag/report_flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ var (
Name: "ignore-policy",
ConfigName: "ignore-policy",
Default: "",
Usage: "specify the Rego file path to evaluate each vulnerability",
Usage: "specify the Rego file path (or dir path with Rego files) to evaluate each vulnerability",
}
ExitCodeFlag = Flag{
Name: "exit-code",
Expand Down
53 changes: 49 additions & 4 deletions pkg/result/filter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@
import (
"context"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"

"github.com/open-policy-agent/opa/bundle"
"github.com/open-policy-agent/opa/rego"
"github.com/samber/lo"
"golang.org/x/exp/maps"
Expand All @@ -14,6 +17,7 @@

dbTypes "github.com/aquasecurity/trivy-db/pkg/types"
ftypes "github.com/aquasecurity/trivy/pkg/fanal/types"
"github.com/aquasecurity/trivy/pkg/log"
"github.com/aquasecurity/trivy/pkg/types"
"github.com/aquasecurity/trivy/pkg/vex"
)
Expand Down Expand Up @@ -66,14 +70,37 @@
filteredLicenses := filterLicenses(result.Licenses, severities, opt.IgnoreLicenses, ignoreConf.Licenses)

var ignoredMisconfs int
if opt.PolicyFile != "" {
if opt.PolicyFile != "" && len(filteredVulns)+len(filteredMisconfs)+len(filteredSecrets)+len(filteredLicenses) > 0 {
log.Logger.Debugf("Filtering result with ignore policies, type: %s, path: %s", result.Type, result.Target)
var err error
var ignored int
filteredVulns, filteredMisconfs, ignored, filteredSecrets, filteredLicenses, err = applyPolicy(ctx, filteredVulns, filteredMisconfs, filteredSecrets, filteredLicenses, opt.PolicyFile)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just subsequently exec this function for each target ignore rego file (either specified or found in the folder if the input value is the dir path) in the loop below.


// If the PolicyFile option is a dir find and apply rego files in it
var policyFiles []string
fi, err := os.Stat(opt.PolicyFile)
if err != nil {
return xerrors.Errorf("failed to apply the policy: %w", err)
return xerrors.Errorf("failed to analyze ignore policy %s: %w", opt.PolicyFile, err)
}
if fi.IsDir() {
policyFiles, err = findPolicyFiles(opt.PolicyFile)
if err != nil {
return xerrors.Errorf("failed to find policy files in %s: %w", opt.PolicyFile, err)
}
if len(policyFiles) == 0 {
log.Logger.Warnf("No ignore policies found in %s", opt.PolicyFile)
}
} else {
policyFiles = append(policyFiles, opt.PolicyFile)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd like debug output to show which policy files it found when --debug is used so I can verify it is seeing the policy files I think it should load (or not)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. And I was using exactly that debug logging during development 😄
The only inconvenience is that the ignore policies are loaded and applied for every result filtration.
So, I've added the logging of what result is being filtered and then the applied policies.

...
2024-02-15T13:51:33.946+0200	INFO	Detecting pip vulnerabilities...
2024-02-15T13:51:33.946+0200	DEBUG	Detecting library vulnerabilities, type: pip, path: requirements.txt
2024-02-15T13:51:33.949+0200	DEBUG	Detecting library vulnerabilities, type: pip, path: api_client/requirements.txt
2024-02-15T13:51:33.949+0200	DEBUG	Detecting library vulnerabilities, type: pip, path: dbt/requirements.txt
2024-02-15T13:51:33.949+0200	DEBUG	Detecting library vulnerabilities, type: npm, path: engprod/package-lock.json
2024-02-15T13:51:33.951+0200	DEBUG	Detecting library vulnerabilities, type: pip, path: misc/monitoring/searchblox/requirements.txt
--- NEW ---
2024-02-15T13:51:33.968+0200	DEBUG	Filtering result with ignore policies, type: pom, path: node_modules/serverless/docs/providers/openwhisk/examples/hello-world/java/pom.xml
2024-02-15T13:51:33.968+0200	DEBUG	Applying ignore policy: engprod/only_package.rego
2024-02-15T13:51:33.970+0200	DEBUG	Applying ignore policy: engprod/subdir/flask.rego
2024-02-15T13:51:33.972+0200	DEBUG	Filtering result with ignore policies, type: pom, path: node_modules/serverless/lib/plugins/aws/invoke-local/runtime-wrappers/java/pom.xml
2024-02-15T13:51:33.972+0200	DEBUG	Applying ignore policy: engprod/only_package.rego
2024-02-15T13:51:33.973+0200	DEBUG	Applying ignore policy: engprod/subdir/flask.rego
2024-02-15T13:51:33.974+0200	DEBUG	Filtering result with ignore policies, type: npm, path: package-lock.json
2024-02-15T13:51:33.974+0200	DEBUG	Applying ignore policy: engprod/only_package.rego
2024-02-15T13:51:33.975+0200	DEBUG	Applying ignore policy: engprod/subdir/flask.rego
2024-02-15T13:51:33.976+0200	DEBUG	Filtering result with ignore policies, type: pip, path: requirements.txt
2024-02-15T13:51:33.976+0200	DEBUG	Applying ignore policy: engprod/only_package.rego
2024-02-15T13:51:33.977+0200	DEBUG	Applying ignore policy: engprod/subdir/flask.rego

}

for _, policyFile := range policyFiles {
log.Logger.Debugf("Applying ignore policy: %s", policyFile)
filteredVulns, filteredMisconfs, ignored, filteredSecrets, filteredLicenses, err = applyPolicy(ctx, filteredVulns, filteredMisconfs, filteredSecrets, filteredLicenses, policyFile)
if err != nil {
return xerrors.Errorf("failed to apply ignore policy %s: %w", policyFile, err)
}
ignoredMisconfs += ignored
}
ignoredMisconfs += ignored
}
sort.Sort(types.BySeverity(filteredVulns))

Expand Down Expand Up @@ -223,7 +250,25 @@
}
}

func findPolicyFiles(policiesDir string) ([]string, error) {
var files []string
err := filepath.WalkDir(policiesDir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if !d.IsDir() && filepath.Ext(path) == bundle.RegoExt {
files = append(files, path)
}
return nil
})
if err != nil {
return files, xerrors.Errorf("walk error %w", err)
}

return files, nil
}

func applyPolicy(ctx context.Context, vulns []types.DetectedVulnerability, misconfs []types.DetectedMisconfiguration, scrts []ftypes.SecretFinding, lics []types.DetectedLicense,

Check failure on line 271 in pkg/result/filter.go

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest) 

tooManyResultsChecker: function has more than 5 results, consider to simplify the function (gocritic)

Check failure on line 271 in pkg/result/filter.go

View workflow job for this annotation

GitHub Actions / Test (ubuntu-latest) 

tooManyResultsChecker: function has more than 5 results, consider to simplify the function (gocritic)
policyFile string) ([]types.DetectedVulnerability, []types.DetectedMisconfiguration, int, []ftypes.SecretFinding, []types.DetectedLicense, error) {
policy, err := os.ReadFile(policyFile)
if err != nil {
Expand Down
Loading